A brief recap:

1. I added a blog widget that automatically links similar posts on my site.
2. I said, "gee, it'd be neat to link to the rest of the web this way" and submitted a feature request to the ether.
3. I ingested a bunch of RSS and listed blogs with post titles/descriptions similar to my tag list.
4. Since tags, titles, and descriptions aren't much data, I started estimating similarity based post text. <-- You are here.

It's all chronicled by my meta tag, but the tldr is that I have my own static site generator with markup that I use to write posts. My Element abstract base type is inherited by text sections, images, galleries, tables, etc. So it was pretty straightforward to add getPlaintext() to this type and let the child classes decide what would go into a tokenization.

Hitting the markup was considerably easier than trying to tokenize the output html. And so each post object now has:


It's easy to find generic lists of stopwords ('a', 'the', 'therefore'...), I tailored mine by running getTokens() on every post and scrutinizing the top of the histogram.

So my earlier post recommender code became:



Since 'cowboy' means one thing, 'bebop' means another, and 'cowboy bebop' means yet another, I used words (1-grams), 2-grams, and 3-grams for the tokenize() operations mentioned above. Dumping the token intersections (above and below) shows reasonable results: most of the words have significance and can indicate two posts are alike.


Indeed, tags are really good data that shouldn't be ignored just because I have a more expansive dataset. While the tag dictionary is only a few hundred words long, they are words chosen specifically to label the subject matter of the post.

Plus the code is already there. So, what, like weight them 50/50?

I dumped some example data, here is the similarity calculator reporting new high scores as it iterates through all the other posts:


Happily, the values diverge quite a bit, so this isn't just a more computationally-intensive way to do what I was doing with tags. But since the tag similarity values tend to be an order of magnitude greater than the token ones, naively adding the metrics would give the tags a lot of weight. With a little extra code, I normalized each measurement (with its own maximum) and then weighted them equally.

Source.

Finding similar posts on other websites is conceptually the same as this but since it requires a lot of automated page visits, it's a post for another time. But I still have my rss/xml data that I previously compared to kilroy tags. Some of the token-based results:

Latest post	Similarity (pct)	Simliarity (nom)
wizardzines.com	0.113	817
mooreds.com	0.108	369
jhey.dev	0.107	112
zander.wtf	0.106	236
tonmeister.ca	0.099	453
vianegativa.us	0.098	499
usuf.fyi	0.097	370
engineersneedart.com	0.097	144

Clicking through them, it's not exactly like looking in a mirror, but then a lot of the match tokens looked like:


While 'game' and 'meme' aren't bland enough to qualify as stopwords, there may be more significance to matches on uncommon tokens/n-grams. That may something to add to the backlog, more immediately I need to grab the actual posts rather than rely on the descriptions.

For now, here's a fun cheat sheet from the list:

Source.

Related - internal

Related - external