Tell HN: 6.3% of HN top submissions in plain HTTP, more than half upgradable

I was using the HN front page to test a library I was writing when I noticed some links that probably should be HTTPS are in plain HTTP. This piqued my interest a bit so I did a little analysis on how prevalent plain HTTP links are on HN. I probably don't need to rehash the harm of using plain HTTP, even for personal blogs -- they can be snooped, and they can be modified to inject either ads or more sinister payloads. In fact, years ago I once disabled my ad blocker by accident and saw an ISP-injected ad on my personal site; never again, I swore.

The methodology is simple. I gathered all links from https://news.ycombinator.com/front ("past" on the navigation bar) for each day from 2020-01-01 to 2020-07-09. These are the top stories of each day. This is a trivial task and resulted in 17566 links (raw data [0][1][2]). There are <100 duplicates, which I kept. Among these are 1112 plain HTTP links, amounting to ~6.3% out of 17566.

Next I analyzed how many of the 1112 plain HTTP links are available over HTTPS. Methodology:

1. Check if the HTTP version redirects to the HTTPS version; if so, done, otherwise record the HTTP response;

2. Replace http:// with https:// and see if the HTTPS URL works; if so, record the HTTPS response;

3. Compare the HTTP and HTTPS responses. If they're identical, done. If not, compare the length of the responses; if they differ by <=1%, record this as HTTPS response almost identical as HTTP, and assume the HTTPS version works (the page may not use relative URLs or omit the protocol, so the HTTPS response may be subtly different while having the exact same rendered output).

The analysis script is available at [3].

---

To be continued in a comment since I'm hitting the 2000 char limit.