Site-details – Journal of PlanetMike

No Advertising on PlanetMike.com

Michael Clark — Thu, 24 May 2012 12:54:07 +0000

What the heck is wrong with the idiots out there that may think I want to run advertising on this web site? Do you see any advertising on the site already? I realize they’re just sending spam, and are just mass-mailing everyone online in the hopes of getting their drivel out there. But geesh, people are idiots. Both the spammers, and the fools that fall for it. 7b1d91231a87fb75e0054e886a0dea57

A real advertising offer will have a web site included in the message. The return address will not be a disposable domain name, like gmail.com, hotmail.com, or yahoo.com. The message will also follow the CAN-SPAM law. There will be a postal mailing address in the message. There will be an opt-out link. (Although you should never opt-out of spam in the first place since all that does is confirm your email address is being read).

So in case you’re not clear: I am not interested in running your text ad, or banner ad, or other wonderful advertising solution on this site. Go away.

There. I feel better now. I just wanted to get that off my chest.

198.136.50.162 06/Dec/2019:01:57:36

Copyright © 2019 Journal of PlanetMike. This Feed is for personal non-commercial use only. If you are not reading this material at http://www.planetmike.com/ or in your news aggregator, the site you are looking at is guilty of copyright infringement. Please contact copyright@planetmike.com so we can take legal action immediately.

Using 301 vs. RewriteRule

Michael Clark — Mon, 22 Jun 2009 13:00:58 +0000

A few weeks ago, I wrote out a series of steps aimed at Fighting Bots Via Their Bad Requests. After watching my logs since then, I’ve noticed I made an incredibly stupid mistake. Bad bots do not follow 301 redirects! What does that mean? 7b1d91231a87fb75e0054e886a0dea57

If a regular browser encounters a web page, image, or document and is told by the web server that that item has been moved (via a 301 response code), the web browser will then ask for the item at its new location. That is the redirect. But it is up to the browser to ask for the item at its new location! Bad bots don’t care if the item has been moved, they’re just looking for vulnerabilities to exploit.

So I have removed the R=301 code from my redirect requests. And now if a bad bot asks for something like http://www.planetmike.com/2007/02//include/scripts/export_batch.inc.php?DIR= they will not be redirected to my robot-trap, but instead will be given the robot-trap! Poof! Instantly they are blocked. I was making things more complicated than they needed to be.

Fighting Bots Via Their Bad Requests

Michael Clark — Sun, 07 Jun 2009 13:45:48 +0000

Last week I started looking through my “page not found” (404) errors. It’s been interesting to say the least, with nearly 800 bad requests since then, most of which have been various attacks in trying to mislead people. I address most of these attacks or probes by re-routing the request to a robot block system based on Daniel’s Webb’s Bot-trap – A Bad Web-Robot Blocker. In the code blocks listed below, I have listed a sample of the commands I have entered into my site’s .htaccess file. Warning: Messing with your .htaccess can break your web site. Be Careful! 7b1d91231a87fb75e0054e886a0dea57

Scanning for Remote File Include Vulnerabilities. There are many bugs in many diferent programs out there on the web. So when someone scans my server for oen of these things, they aren’t up to any good. Common requests include errors.php, contact.php, advanced1.php. There is (was) a compromised server at
- http://www.eyepro.net//assets/images/id1.txt
- http://www.eyepro.net//assets/images/master-id.txt
- http://www.graal-plus.zp.ua//images/roxx.jpg
- http://grupowh.com/sqli/fx29id.txt
- http://www.ecobook.or.kr/ecobook/data/ecobook/1132289642/copyright.txt
- http://www.centrsoft.ru/logo.jpg
- http://www.cookieez.com/image.jpg
- http://largeface.com/gnuboard4/style/sid.txt
- http://harvestusa.org///administrator//includes/id1.txt
Someone at these IP addresses was scanning:
- 80.67.20.178
- 62.193.227.12
- 212.193.241.25
- 70.245.218.25
- 203.185.28.194
- 200.30.136.59
- 74.55.117.34
- 74.36.117.160
- 94.23.200.54
- 211.189.18.73
These probes are now being dealt with in real time.
See Fx29ID cmd for a little more information.

RewriteRule errors.php /robot-trap/ [L]
Requests for my favicon.ico in a directory other than the root. The only thing in common with the requests is they are all using GoogleToolbar 6.1; Windows 6.0; MSIE 7.0. This is a silly request, is there ever a reason for a favicon to be anywhere other than the root of the site? I’ve now dealt with these as well.
RewriteRule .+/favicon.ico /favicon.ico [L]
There were a very few requests for a malformed URL that was actaully part of my server’s file system. That’s been fixed.
RewriteRule var/www/html / [L]
I discovered a Java-based browser (probably a spam bot seeking email addresses) that kept tripping one URL that wasn’t being rewritten correctly. So I fixed my rewrite rule.
- 213.93.203.217, Java/1.6.0_04
- 24.132.227.22, Java/1.6.0_13
- 77.211.115.58, Java/1.6.0_04
- 82.192.63.216, Java/1.6.0_13
- 84.124.194.76, Java/1.6.0_13
Now to decide how to deal with these “web browsers.” Based on the traffic activity (one request every 1-2 seconds, no images or style sheets requested at all) should I even allow these browsers to access my web site? I’ve also decided to block access to my web site by Java user agents. See How To Block Java User-Agents for someone else’s similar approach to the Java problem.

RewriteCond %{HTTP_USER_AGENT} Java.* RewriteRule ^(.*)$ /robot-trap/ [F]
For some bizarre reason, the MSNBot (shouldn’t that be the BingBot now?) is sending requests including an anchor. Their requests are showing up as “GET /url-stuff-in-here/#respond HTTP/1.1”. I can’t see how the hash symbol (pound symbol) is being logged at all, I can’t reproduce the problem. Helpfully, MSN does include with their browser agent, a URL that can be used for help. “msnbot/2.0b (+http://search.msn.com/msnbot.htm)”
Just try visiting that msn.com URL. You get bounced over to a site at live.com, which is now Microsoft’s bing search engine. Using Firefox 3, OmniWeb, or Opera, after “signing in” I kept getting into a loop where it would ask me to join the community. I couldn’t get past that point. I finally fired up Virtual Box and used IE8 under Windows 7. Sigh. And Microsoft wonders why people loathe them so. (Remember Bing stands for But It’s Not Good.)
Speaking of Microsoft, some poor souls are still using IE 6, with the Discussion bar turned on. Discussion bar requests (/_vti_bin/ or /MSOffice/) are now being redirected to the I6 page at BrowserUpgrade.info.
RewriteRule _vti_bin http://www.browserupgrade.info/ie6/ [R=301,L]
A few web browsers were badly broken, and were requesting things like http://www.planetmike.comhttp://www.planetmike.com/2006/08/. I fixed this.
RewriteRule www.planetmike.com(.*) $1 [R=301,L]
A few bots were attempting to poison my referral logs, or to otherwise do Bad Things by requesting multiple times files on my site with a space in the file name. And to make things look legitimate, they set the referral to be a search at Google.
RewriteCond %{HTTP_REFERER} google\.com RewriteCond %{REQUEST_URI} .*\ .* RewriteRule ^.*$ /robot-trap/space.php [L]
A few requests for labels.rdf. This is a standard for labelling how family-friendly your web site is. See the Family Online Safety Institute Labelling Page for information. This is the next generation site labelling method after the PICs method which was used in the ’90’s. For now I’ll let these requests return a 404.
Bad BrowserAgents. No browser agent at all? Denied! An agent of “anonymous” is not allowed either. Especially when your IP address is registered to Korea or Brazil.
BrowserMatchNoCase "^$" spambot=1 BrowserMatchNoCase "^anonymous$" spambot=1 Order deny,allow deny from env=spambot
And there have been a few links on my own site that were misspelled, or had other stupid mistakes. Those have been fixed.

This is a start, I’ll post updates as more unique cases appear.

Journal of PlanetMike

Evolving PlanetMike.com: Chapter 5 – New Name

Michael Clark — Thu, 07 Feb 2008 21:30:09 +0000

Currently most, and soon all, of the content at PlanetMike.com is related to technology and web design. So I took this opportunity to rename the site from “Michael Boyd Clark Journal/Blog” to “PlanetMike’s Technology Journal.” I am now adding new GoogleAlerts to monitor for the new name appearing in splogs (already found two!). I also tweaked the settings in my rss footer plugin. 7b1d91231a87fb75e0054e886a0dea57

Evolving PlanetMike.com: Chapter 4 – Copying the Database and Redirecting

Michael Clark — Wed, 06 Feb 2008 23:03:18 +0000

Ok, I’ve moved a whole bunch of posts from PlanetMike.com over to my personal site, MichaelClark.name. While tedious, it was a pretty straightforward process. 7b1d91231a87fb75e0054e886a0dea57

Install WordPress on your new domain. Make sure it is the same version as the one on the old domain!
Copy the active theme to the new site’s theme directory. This should be under wp-content/themes/.
Copy all of your plugins to the new site’s plugins directory. This should be under wp-content/plugins/.
Copy the WordPress database to a new database. Ideally, make a backup of the old site’s database, and then import it into a new database. At this point you have two WordPress database set up with 100% duplicated content, users, passwords.
Use a mysql editor to edit the “siteurl” and “home” values in the wp_options table. I used Navicat, but you could use any mysql database editor like PHPMyAdmin or the command line.
Then simply edit the wp-config.php file on the new site so it can read the new database.

Tada! You now have a complete copy of your old WordPress blog on a new domain.

I took this opportunity on the new site to change my permalink structure. I removed the dates from my post’s URLs.

Important! If you have any photos, style sheets, images, included files, or other content referenced from the old site, those links won’t work on the new site until you copy or move that content over. And since I was splitting out my personal information, I had a lot of other info to move around.

My goal was to have my photos, family stuff, jokes, etc… all on my personal site. So I moved those directories to their new location. But for now I don’t want those pages to break if someone visits that URL. So I created a link from the new site to the old site. For example:
ln -s photos ../www.planetmike.com/photos

I did this for a bunch of directories. To the end user, you can’t tell that anything has changed. But I do have the same content on two domains now, which could mean a penalty at some point from search engines. The reason I had a bunch of directories is that PlanetMike.com has been around since 1997, the days before there were content management systems available.

Now another tedious step: removing duplicate content from the blogs. On the new site I went through and removed a bunch of posts and categories that will not be used there. On the old site I didn’t have to do anything with those posts, since they aren’t changing.

And the final tedious step: On the old site, look at each post in the categories for the new site. A very few of these posts could be deleted entirely, from both blogs. But most needed to be redirected to the new site. I created a list of URLs from the old site that were to be redirected to the new site. I did this by category, just to keep this in manageable chunks. A few posts I reclassified into other categories. Then in my .htaccess for the blog, I added the RewriteRule, one for each URL:
RewriteRule 2005/09/15/virginia-tech-photographs/? http://www.michaelclark.name/virginia-tech-photographs [R=301,L]

This rule says if the requested url contains 2005/09/15/virginia-tech-photographs (with an optional slash at the end), that page has been permanently redirected to http://www.michaelclark.name/virginia-tech-photographs. As I added each category’s URLs to the .htaccess file, I deleted the post from the old blog. Anyone that requests the old URL would transparently be bounced to the new site. That might be confusing to people who visit the site during the transition.

I ended up with over 270 lines in the .htaccess file. I could have used some regex (regular expression) magic, but I liked actually touching each URL. By doing each URL one at a time, I was able to basically audit the entire site.

I finished this step a week ago, and have been watching my server logs and my Google webmaster account for problems. One problem I’m seeing now is that some of my category and monthly archives had multiple pages, but since I’ve pruned the number of posts down, the highest numbered pages don’t exist any longer. I’ll be leaving that as a 404 error so that the search engines will remove that page from their indices.

Evolving PlanetMike.com: Chapter 3 – Moving Content Between Domains

Michael Clark — Thu, 10 Jan 2008 12:48:00 +0000

I’ve made a good start on splitting out personal stuff from PlanetMike.com, and moving it to MichaelClark.name. Jokes, photo galleries, and a handful of categories from WordPress have been moved. I’m moving a few more categories today, then I’ll share some of the lessons I’ve learned. Hopefully you won’t see too many problems on any of the pages during this transition. I’m using linked directories, and apache’s rewrite rules to gracefully move content around. 7b1d91231a87fb75e0054e886a0dea57

Usage Statistics for PlanetMike.com: 2007

Michael Clark — Wed, 02 Jan 2008 03:48:08 +0000

7b1d91231a87fb75e0054e886a0dea57


Summary by Month

Month	Daily Avg				Monthly Totals
Month	Hits	Files	Pages	Visits	Sites	KBytes	Visits	Pages	Files	Hits

Dec 2007	15,151	13,329	8,748	2,681	42,654	7,180,572	83,124	271,191	413,207	469,697
Nov 2007	15,551	13,817	8,359	2,690	43,339	7,285,245	80,720	250,796	414,536	466,559
Oct 2007	14,355	12,382	7,932	2,829	41,912	5,712,332	87,721	245,918	383,861	445,030
Sep 2007	16,362	14,760	8,595	3,121	44,531	6,048,530	93,651	257,851	442,803	490,879
Aug 2007	19,132	17,161	9,987	3,957	53,589	9,012,449	122,670	309,605	532,000	593,098
Jul 2007	19,288	16,980	10,382	4,510	54,278	7,551,001	139,820	321,848	526,400	597,931
Jun 2007	19,756	17,743	10,586	4,326	55,907	5,610,407	129,796	317,585	532,290	592,688
May 2007	19,309	17,420	10,541	3,538	51,775	5,939,830	109,689	326,784	540,050	598,607
Apr 2007	23,801	21,295	11,800	3,880	62,231	7,530,408	116,417	354,002	638,869	714,044
Mar 2007	18,459	15,847	10,354	3,159	46,489	6,788,780	97,932	320,974	491,286	572,233
Feb 2007	20,256	17,948	9,690	3,415	57,104	6,743,685	95,634	271,332	502,559	567,168
Jan 2007	21,433	18,710	11,511	2,954	46,985	6,808,148	91,595	356,869	580,011	664,446

Totals						82,211,387	1,248,769	3,604,755	5,997,872	6,772,380

Evolving PlanetMike.com: Chapter 2 – Where will it go?

Michael Clark — Thu, 20 Dec 2007 18:03:46 +0000

I’ve said the personal posts and content from PlanetMike.com will be moved to another site. What site would that be? I’m thinking now it will be MichaelClark.name. I also have registered both MichaelClark.org and MichaelClark.net, but .name will be my “official” personal site. 7b1d91231a87fb75e0054e886a0dea57

If you’d like to get a glimpse as to how I’ll be doing this, simply subscribe to my personal RSS feed at http://www.michaelclark.name/feed.

Evolving PlanetMike.com: Chapter 1 – What do I currently have?

Michael Clark — Wed, 19 Dec 2007 12:33:15 +0000

So the first thing to do is figure out exactly what content I have to work with. For the WordPress portion of the site, that’s simple. I installed the Dagon Design Sitemap Plugin (warning – lots of spammy comments, so that site may not be managed any longer) and created a sitemap of the blog portion of PlanetMike.com. 7b1d91231a87fb75e0054e886a0dea57

I also simply eyeballed the site’s home page of PlanetMike.com to see how many sections of the site could be rolled over to my new personal site. It’s a lot to move over: photography, fractals, all of the Humor sections (jokes, optical illusions, and funny photographs (which may simply get deleted since I don’t think I own the copyright on any of them)). There are also a lot of my writings from before there were blogs that will be moved into the blog system. That will also make the site easier to navigate and maintain.

Ultimately my final goal is to have everything in the WordPress database. That may not be possible, but as far as I can do it I will. For example, my web traffic log reports output into shtml (server side include) files. I’d rather they were in my WordPress database. Can I copy and paste the tables and reports into a new page in WordPress? A little extra work, but that’s ok. Can I do the same thing with my Gallerie photo galleries? I think I’ll probably look around at the options available for photo management plugins for WordPress.

Evolving PlanetMike.com: Introduction

Michael Clark — Tue, 18 Dec 2007 17:56:50 +0000

For years I’ve been using PlanetMike.com as my primary web site, and I post everything on it. The site started as a bunch of static web pages, then used (and still does in many places) server side includes. I’ve posted photos, jokes, other miscellaneous information. Then I started blogging, using Bloxsom, and then switched to WordPress. My category list is enormous. 7b1d91231a87fb75e0054e886a0dea57

Well, I am now getting ready for the next step in the evolution of the PlanetMike.com web site. I am going to move the personal stuff off of PlanetMike.com over to another domain. I want PlanetMike.com to be a site that a new blogger (or an experienced blogger) can come to to learn about technology and related issues. And obviously a lot of the content on the current site is a distraction and clutters the message that I am a professional who knows what he’s talking about. So I’m going to streamline the site, move content around (e.g. the blog will be the top level, the journal directory will be no more), and end up with two sites: one covering technology and blogging issues, and one covering me and my life.

I plan to start this process shortly after Christmas, and being finished up sometime in January. I’ll be writing about my experiences in a series of posts here on PlanetMike, since I’m sure I will learn things that others will need to learn as well. A key part of this is to not break old URLs and links, like I wrote about a few days ago in Help Your Users: Don’t Break Old URLs.