A few months ago Steve and I noted that our respective sites were getting tons of hits from Samara Oblast, an obscure(?) territory in Russia. Russian search engine maybe? Cybercriminals? Proxy for the American or Chinese or Syrian electronic armies? Who really cares? Only port 80 should be open and doing nothing fancy

But since this kilroy thing has gotten pretty lengthy I was scoping the possibility of doing some sort of 'top content' thing based on hits. So I pulled my server logs and was looking through them to see how hard it'd be to parse.

Source.

Well this is fun:


How am I going to count hits for 2008/01/index.php when there is no anything.php?

Eight sequential hits from the same person, within 10 seconds. That's what I call quick on the mouse. Whois says it's from Ukraine. I'm going to stop me right here, this is my first time actually looking at http traffic, this is old hat to 80% of the world. Okay, let's continue.

• First they hit an index page that actually works.
• Then another page which may or may not be linked from the first page.
• Now six sequential hits looking for index.php in various legitimate and illegitimate directories.

Maybe they're just guessing about site map, but probably they're looking to have some fun with php.

Another interesting one:


Looking to do injection or overflow or something? Not really my wheelhouse, but it was kind of a fun digression.

So I wrote some code to classify site traffic into one of the following categories:

• Attacks: malicious traffic like the stuff depicted above.
• Bots: googlebot, baidubot, hedonismbot, etc.
• Visits: everything else, I assume, is either a person with a browser or a bot trying to emulate one. They are probably equally entertained.

Some of it was pretty easy, bots tend to declare themselves in the user agent string and hit robots.txt first. Malicious stuff sends PUTs and looks for files that aren't .html/.jpg/etc. And, of course, sequential traffic from the same IP can be classified together. This is important because an attack might hit numerous legit links but it's not visit traffic.

Logs go back about a year. Here's some excel because easy.


I get indexed about twice as much as I get visited. There have been more than 20,000 malicious http requests.


Google, Baidu, and Majestic 12 (a distributed indexing project) turned up most. But there are quite a few bots out there.

So the top visited content, the main reason for this whole endeavor:

Pages

• East coast, Friendsgiving.
• April 2008; surfing, Speed Night, autocross.
• October 2014; surf shots, footgolf, a dog on a hill.
• The January 2008 Speed Night rankings.
• January 2015; scuba, surfing, SimCity, snow.

Images

• Erik getting closed out in the dark, and one more from that session.
• Scott with some light tracers, and three more from that session.
• Derrick facepalming at Del Mar, and 20 or 30 more from that session.
• Triptych of Kafka chomping a treat.
• A guy paddling out at Pipeline.

Labels - which are now just links to search

• Media room
• A&A:AE
• Shower
• Paint
• Lighting

Data skew: some content has been around longer. On the other hand, the logs are only from about a year back.

When I get some more fun-coding time I'll see about putting this in the sidebar.

Related - internal

Related - external