Make search your own!

Internet search sucks. It's not just my personal opinion.

If you don't trust the media, maybe you do trust the Germans: their university researchers came to the same sad conclusion “that higher-ranked pages are on average more optimized, more monetized [...] and they show signs of lower text quality”, reads their recent paper.

Maybe you recognize this from recipe websites. While the recipe is the part you are really interested in, at some point in history recipe pages started to bloat. Now you need to plow through a pile of text and pictures before you can even take a look at  the recipe itself. This additional clutter helps individual recipe pages rank better in search results: "When I’m writing, I try to tell a story that has a hook as well as please[s] the Google algorithm,", says this food blogger. But what Google wants to show is not necessarily what everybody wants to read. Such ‘optimisation’ then gives birth to projects like justtherecipe.com - a website and app that only extracts the recipe from any given website.

Pessimists say that the internet is only going to get more cluttered. With the dawn of generative AI, we will swim in more and more low quality content. The internet will never be the same as it was when populated just by us  - the humans. And as much as I love robots, they also produce text that can contradict itself in consecutive sentences;  in a much worse case it can make you poison yourself with mushrooms. The bots are also capable of making us forget what a baby peacock looks like.

So how do we search for high quality content in waters made murky by SEO optimization and generative AI? For me a  search engine is like a map, and I have to keep reminding myself that like a map it only ever shows part of the landscape.

Google has been steadily holding about 90% of the search engine market for at least the last 15 years. But the good news is you don’t have to keep on only Googling. The key to any healthy diet is variety. So take my hand into the (expanding) possibilities of searching the web.

Every day I use a couple of different search engines. All of them are designed with a different approach to indexing the web.

Keyword search

Ecosia

Ecosia is my default search engine. I use it when I know what I am searching for: company names, people, wikipedia articles, documentation. Under the hood it uses two of the big boys : Google and Microsoft Bing. The search is therefore still not better than Google, but their business is. Ecosia is the largest not-for-profit search engine that puts 100% of their revenues into planting trees.

As well as search, Ecosia also offers a free AI chat feature, based on GPT-3. This one I use for translations and when I need definitions of some terms or phenomena. I also use it for basic recipes, and tweaks for recipes (especially for baking - which I still find mesmerizing).

Google

Google is not completely out of my system, but I currently use it only when I can not find things via Ecosia. Google is amazingly good at finding out where to buy things online, so I do my infrequent online shopping ventures on Google.

Marginalia

Marginalia is what I like to see as a serendipity search engine: you never know what you will get. The index is built by Swedish developer, Viktor Löfgren, mostly on grant money (he also has a patreon). It  focuses on “small obscure websites and non-commercial content”, as the author puts it. 

What I love about Marginalia is that when I searched for a recipe for a crumble, instead of one of the sterile over-optimized recipe pages, I discovered the Kitchen Projects substack, where somebody called Nicola Lamb documents their experiments in crumble baking. Next to the recipe, she also shares explanations of how baking with fruit works. If this doesn’t give you warm fuzzy feelings for the internet, I don’t know what will.

I use Marginalia if I am interested in anything related to craft (cooking, basket weaving) or if I feel like online exploration outside the social media ecosystem. Their rainbow ‘random’ button is perfect for that.

Embeddings search

LLMs are also taking over internet search. There are new search engines that let you chat with the search results and even formulate new questions based on the previous search.

Two examples of these are Exa and Perplexity. I have discovered them only recently but use them regularly to discover material around a certain topic for my work related searches. 

Perplexity

Perplexity lets you chat with the internet. The interface is very user friendly and includes all the search results the robot is basing its answer on. But be careful here, because LLM models do hallucinate and as WIRED reported: “it falsely claimed that WIRED had reported that a specific police officer in California had committed a crime”. It is therefore more suitable for general overview questions rather than in-depth search.

Exa

Where perplexity makes you chat with the internet directly, Exa returns actual search results first, that you can then chat with if you like, so resembling an actual search engine more closely. It also (automatically) alternates between keyword search and embeddings-based search depending on what you ask. This is a good call, as keyword search is less thirsty for resources. I also like Exa’s focus on using LLMs to return relevant search results.

Premium

Kagi

Kagi is the only search engine in the list that you need to pay for - if you want to do more than 100 searches a month. In return they won’t run ads or sell your data. You can personalize their keyword search;  they also offer an AI bot feature where you can choose from a dozen LLMs . 

As I don’t have premium access, I use Kagi sparingly, for the most in depth part of any research. The search results are very good and it helps me find things that I could not find using any of the other search engines.

—

This list is personal, and by no means exhaustive. I for example don’t use the most famous privacy oriented alternative to Google, DuckDuckGO. There also exists a little army of small ‘folk’ search engine like Marginalia. This list is very much a work in progress and a living active process. 

The Internet morphs and changes. Looking ahead to 2034 the internet will be at least as different from now as now is different from how things were in 2014. Therefore we need to keep our strategies for navigating it flexible too.

Have other search engines you use and love? I would love to hear about them!