Modeling trust on the AI-slop web
I am currently building a search engine. It seems a weird choice, and maybe unnecessary? So let me explain why I am doing this.
The Internet has been overrun by automated content creation machines. I guess because people and businesses have FOMO about appearing in search results (SEO) and LLM outputs (GEO? AEO?) and they want to sell their products and services. All that AI generated stuff is overwhelming the human-made Internet. Putting text on the internet is no longer proof-of-work. Because of this it has become very difficult for me to find good information. So I've been reading up on ways to deal with that.
I know that Jason Kottke has mentioned the same problem. Blogrolls and Webrings are making a comeback, because bloggers see the added value of referring readers to other good websites. Apart from generated text, there's a problem with generated code, too. The open source community has to deal with a firehose of AI-made PRs. Mario Zechner, the creator of agent harness pi, gave a good keynote about this at AI Engineer Europe 2026. Pydantic's Samuel Colvin proposed a federated reputation system for open source contributors during PyAI 2025.
¶Modeling trust
I particularly like one concept from information security (cryptography), which is called Web of Trust. The idea is that you can make a network, a graph, that describes a system where people vouch for each other. So the graph consists of nodes and directed edges. Nodes are cryptography keys, and by proxy, people. An edge from A to B says "The person with key A signed key B". Signing someone's key means "I know that this key actually belongs to person B". If enough people vouch for a group of other people, you get a graph that covers a lot of people. If you trust the vouching method, you can walk the graph and find a path of trust from your key to another person's key, and propagate your trust through the network. This is an alternative to having a centralized certificate authority. If you want to know more, read the Wikipedia article.
PGP is built on this idea of vouching. But PGP is very difficult for most people to use, so that doesn't work great in practice. The only people who have signed PGP keys tend to be people that work in information security. Most people outside of the security community will have never heard of this concept.
Mathematicians have something similar (I'm saying this tongue-in-cheek) to model collaborations. Mathematicians have the so called Erdős number, which is your degrees of separation from the famous Hungarian mathematician Erdős. Erdős himself has the Erdős number 0. And then everyone who has ever published a paper together with Erdős has an Erdős number of 1, and everyone who has ever published a paper with one of these people has an Erdős number of 2, and so on.
Fun fact: my Erdős number is 4, because I published a paper together with Herman Geuvers on type theory during my computer science studies. This is only one of the many benefits of a research master in mathematical foundations of computer science. *snickers
Now I would love to be able to browse the internet only by going through a subset of the Internet which was created or written by or vouched for by people that I know and trust. You could take the Web of Trust concept, and replace "person A signed the key of person B" with "person A's blogroll contains the website of person B". Or you could assign all indie bloggers a jd7h-number, or a Kalzumeus-number, or a Simon Willison-number, etc. Of course, co-authoring a paper is a symmetrical relationship, whereas linking on the web is not.
My use case is finding trustworthy information. If my AI startup clients have a problem and they want my advice, it is very important that I can find some high-quality information on that problem, fast. Examples: what is the maturity of this python library, what are the best practices for designing an evaluation for this type of AI system, what is the state of the art in building agent swarms, what is currently the best harness, which are currently the most important early-stage AI investors, et cetera.
I don't have time to comb through thousands of pages of AI slop. I'm pretty good at spotting low quality content, but the problem with AI-generated stuff is that it looks really probable. It looks authoritative, it looks convincing. Most of the time I need to deep-read a text before I can detect it is AI slop. (No, counting the number of dashes is not enough, haha).
One way around this is by letting LLMs do that part for me. I have been an enthusiastic user of tools like Claude's research mode, which is a great way of going through a lot of search results in a short time. However, I've noticed that LLMs are not very good at seeing which web pages are high-quality and which are not. An LLM will happily do a "critical review" of a product while only referencing web pages with marketing content. In the end I have to make this judgment call myself. And if I can limit my search space to at least the domains or parts of the internet that I know that I can trust, that makes my life a whole lot easier.
¶Kagi and Exa
Now my first approach to solving this problem was using Kagi.
Kagi is an alternative to Google. It's a search engine, and instead of an advertising model, they have a monthly subscription. And Kagi has this really cool feature where you can either boost or downgrade specific domains. So for example, if you want to see lots of Wikipedia pages and not so many Quora answers, you can boost Wikipedia and downgrade Quora. So this allows you to tweak the domains that are searched and turn up in your search results.
However, it would be too much work to manually Kagi-boost all the domains that I trust. There's hundreds of them. It's still a small set, but it's too large to input this by hand on Kagi, so this solution doesn't scale.
Another problem is that many people write on platforms. Boosting a platform domain, such as substack or github, doesn't work. I'd want to boost an url prefix, so https://github.com/jd7h or https://arnicas.substack.com. I don't think Kagi supports that.
Secondly, I have been using Exa AI. Exa is a search engine built for usage by LLMs. I have integrated Exa at AI startups before, so I was pretty familiar with it. Exa allows you search via an API, and these search actions can be restricted to up to 2000 domains. So this is a great start, and this is what I'm currently using, but I would like to do something even more fine-grained, which is why I'm building a search engine and a trust graph.
¶Indie web sub-graph
So I would like to be able to say "I trust this group of people on the internet, and I trust the webpages that these people link to too." Most of the people on my "trusted" list are indie bloggers that write about software engineering, AI, startups, science or investing. So basically, I would like to assign some kind of trust score to web pages, based on whether my favorite indie bloggers referred to them.
I was reading up on this topic as to how to model trust and relationships in computer networks and social networks. Of course PageRank is a famous example of an algorithm for modelling authority/importance. This was the algorithm used by Google for their search engine in the late nineties.
I also found Advogato (via Hacker news), a very early social media website, created by security researcher Raph Levien. Advogato had a vouching system that is very similar to the vouching system in public key cryptography. Levien's paper Attack-Resistant Trust Metrics that describes Advogato's trust model is pretty readable!
What I would like to do is now build a search engine that incorporates either oldskool PageRank or Advogato's trust modeling that lets me browse my particular subgraph of the internet so I can find information for my clients faster and with higher confidence.
¶Sentiment analysis on outlinks
A critical reader might at this point say, "Well, if an indie blogger links to a page, that doesn't mean that they vouch for it. It's not similar to signing someone's public key." Yes, I realize that. Outlinks are only a weak proxy for endorsement or trust. I don't think indie bloggers tend to link to malware sites or low-quality content per se, although they might link to a page to mock its contents. So yeah, known limitation.
My first idea for fixing this was doing some kind of sentiment analysis on the context of the link. So use some kind of sentiment analysis model to classify whether the author had a positive, neutral or negative sentiment about the target of the link. I tried this with two approaches:
First off, I did a quick search on model repository HuggingFace to find an open weights sentiment analysis model. In the end I went with cardiffnlp/twitter-roberta-base-sentiment-latest. It's nice and small, only 500 mb, so it can run locally. It's not a perfect fit: it's trained on tweets (not web page text) and it's not multilingual (and I might have Dutch language pages in my data). This model didn't work as well as I hoped. One failure mode I saw in my results: if the link is part of a paragraph that says something with negative valence, then the sentiment analysis model will classify the link's context as negative sentiment. For example, a blogger links to a writeup about a hack, written by a well-known security researcher. The link might be titled "Huggingface systems compromised by LLMs gone rogue!". The sentiment analysis model would classify this link as negative sentiment, so an anti-endorsement, and that was not the correct label, obviously. Most of these "negative" links were actually linking to really high-quality writeups of negative news.
I also tried to classify the link context sentiment with Claude Haiku. I was hoping that Haiku (it being a much larger model) could make this fuzzy distinction better: the valence of the topic/contents is not the valence of the linking itself. However, I saw that Haiku 4.5 still made the same mistake, although less often. Later, I learned that this task has a specific name: (citation) intent classification. It's an important topic within academic research, because you can use this to model why academics cite research papers. Do they cite other work because they build upon it, or because they refute it?
I also think that most existing sentiment analysis models on HuggingFace don't transfer well to my use case, because they were trained on radically different data. Most of them are trained on ecommerce data, so webshop reviews for products, which is very different from text on indie blogs. This might be the reason the classifier doesn't perform well on my problem. Anyway, context sentiment is a weak proxy at best. I don't think an approach based on sentiment analysis will work, unless I train a custom classifier myself (which will automatically be a citation intent classifier).
Even if I want to train a custom classifier, I expect to see a huge class imbalance. 99.9% of outlinks will probably be a positive reference, and only 0.1% will be negative. So maybe the risk of misclassification is negligible?
This is work in progress. A domain-filtered Exa search is fine for now, but I'd rather walk a social graph of indie bloggers and work with more fine-grained prefixes. I'd love to come up with a method that scales and doesn't require lots of hand-labeling. Maybe I should start mining Blogrolls, or implement PageRank.