A Note About the ChipGuide

[Charles Kaplan LM-0115-048]

Several months ago the ChipGuide was performing very poorly and users could not access the website. That initiated a study to determine the cause of the performance problems. Initially, we found that a certain feature being used to display images with the correct aspect ratio was the culprit as it had to load each image to determine its dimensions. That was taken care of quickly, with a temporary solution, at least until the image library is upgraded. Until then, some images may not display correctly.

It was further determined that a lot, perhaps 50%, of the website traffic was from "bots". Bots are programs that visit websites to extract information. Some bots are good, like the GOOGLE bot - a search engine, that catalog website content so that users can easily find it. But there are also bad bots. Some marketing companies run bots to extract website data and then sell it to interested parties. Some bots are out to steal website content. The worst bots are looking for website weaknesses that can be exploited, to be malicious or for extortion.

A website tool is now being used daily to monitor ChipGuide website traffic. Literally 100's of bots visit the ChipGuide daily. Most are benign and do a couple of requests and leave. The bad bots are identified and banned from the website. The list of bad bots grows daily. Some of the worst bots cannot be banned because they don't identify themselves correctly. However, based on analysis of their website requests, for example some are looking for weaknesses in WordPress websites, they can be identified and banned by their IP address.

When you ban a IP address, it bans everyone in that vicinity, and may include some legitimate users. If you get a message "403 - Forbidden", when trying to access the ChipGuide, your IP address was caught up with a bad bot. If that happens, just email me and it can be fixed.

Today's Bot Report
You cannot view this attachment.

[Barry Sherwood R-9037 (admin)]

Hi, Charles.  Thanks for adding those monitors and tools (good stuff), but I don't think you sped up the system at all which makes me wonder whether the bots are actually a/the problem.  My test for the speed of the chipguide lies in running the trademaker feature.  I have 65 chips in my want list and trademaker looks to see if anyone else has any of my 65 chips in their trade list.  It has, and still does, take well over 30 seconds to run.

You and I have already discussed this, but I'm throwing this out to the board in case other techie/nerdy people like myself (and you, if you don't mind the compliment) can offer some solutions.

Server response is primarily based on a) CPU speed, b) available memory, c) fetch time (disk speed, latency, RPMs if traditional HD), and d) programming efficiency.

You mentioned that your current hosting provider offered to upgrade you to a faster, newer platform for the same (or reduced?) monthly cost.  I believe you said they'd handle the migration so you wouldn't have to do anything (except backup the current site, just in case).  What's the progress on that?  My last email to you on the subject was 9/28 in case you need to look for it.

I offered to make a copy of the site to my server, which I wouldn't suggest is the absolute best, fastest on the planet, but it's darn good, response time is phenomenal and there are thousands of websites hosted on it.  If the chipguide performs similarly on my system, then the problem is likely not in the server specs and we should look more closely at the programming and database implementation (see next paragraph).  All I need to help facilitate this is root access.  I will not do anything on your server to cause problems.  I'm a seasoned linux system administrator and know my way around quite well.

If/when we start looking at programming performance, this is where Ross Poppel (and others?) could have a huge positive impact as databases are his jam.  It could be a simple indexing issue.  Or some code that loops unnecessarily.  Or makes calls to routines where the result doesn't get used.  I've got a programming background, and I'll help where I can, but I would certainly defer to Ross' more recent and relevant skills.

I believe that some combination of the above can take the 30-40 second trademaker benchmark and get it down to under 2 seconds.  Let's make it happen.  I'm ready.