Lycos, Excite, AltaVista, and of course Yahoo all were originally web directories of this sort.

Both Wikipedia and my own memory disagree with you about Lycos and AltaVista. I'm pretty sure they both started as search engines. Maybe they briefly dabbled in being "portals".

Main problems are:

1. Link rot

2. Sneakily inserted sponsored links

@ajsadauskas @degoogle I guess the problem though is how you make sure they are actually maintained by a human acting in good faith. The way community Facebook groups meant to be for this kinda thing get spammed by likely fake businesses doesn’t give me hope

@ajsadauskas @degoogle What we need to do is re-visit the GnuPG philosophy of building rings of trust. If one emerges with enough people proven to provide quality aggregators/summarizers then we can start to depend on that, or those.

@ajsadauskas @degoogle Webrings! Bring back Webrings!

I used them and contributed to links as well - it was quite a rush to see a contribution accepted because it felt like you were adding to the great summary of the Internet. At least until the size of the Internet made it impossible to create a user-submitted, centrally-approved index of the Net. And so that all went away.

What seemed like a better approach was social bookmarking, like del.icio.us, where everyone added, tagged and shared bookmarks. The tagging basically crowd-sourced the categorisation and meant you could browse, search and follow links by tags or by the users. It created a folksonomy (thanks for the reminder Wikipedia) and, crucially, provided context to Web content (I think we're still talking about the Semantic Web to some degree but perhaps AI is doing this better). Then after a long series of takeovers, it all went away. The spirit lives on in Pinterest and Flipboard to some degree but as this was all about links it was getting at the raw bones of the Internet.

I've been using Postmarks a single user social bookmarking tool but it isn't really the same as del.icio.us because part of what made it work was the easy discoverablity and sharing of other people's links. So what we need is, as I named my implementation of Postmarks, Relicious - pretty much del.icio.us but done Fediverse style so you sign up to instances with other people (possibly run on shared interests or region, so you could have a body modification instance or a German one, for example) and get bookmarking. If it works and people find it useful a FOSS Fediverse implementation would be very difficult to make go away.

Reddit and Lemmy are supposed to be what you want: link aggregators.

We're supposed to link to sites and pages and people vote on how good they are in the context of the sub community topic.

Of course, then Ron Paul happened, and now it's just memes and Yank politics so... maybe deploy Lemmy and turn off comments.

I'd argue that link aggregators like Lemmy (from which I'm posting o/) are the new world version of that. Link aggregators are human-edited web directories; humans post links and other humans vote whether those links are relevant to the "category" (community) they're in. The main difference is that it's an open communal effort with implicit trust rather than closed groups of permitted editors.

@ajsadauskas @degoogle Since I run a small directory this is a fascinating conversation to me.

There is a place for small human edited directories along with search engines like Wiby and Searchmysite which have human review before websites are entered. Also of note: Marginalia search.

I don't see a need for huge directories like the old Yahoo, Looksmart and ODP directories. But directories that serve a niche ignored by Google are useful.

@ajsadauskas @degoogle i love this idea, i'm going to start my own web directory.

I had this exact thought earlier today. Either curated directories, or a ground-up, vetted search engine that only pulls from pre-screened sources.

And is it time to begin considering what a modern version of those early web directories might look like?

Something like fmhy.net?

@ajsadauskas @degoogle I actually contributed to one! I was a writer at LookSmart for four years; we manually created categories and added websites to then, with short descriptive reviews. Though an algorithm listed more sites below our selections, we could force the top result, eg we'd make sure the most relevant website was the first result of a search on that topic. Old-skool now, but had better results in some ways.

@ajsadauskas @degoogle

It looks like there's a couple projects to continue the directory DMOZ. I hope they're sharing work with each other!

@ajsadauskas @degoogle DMOZ was once an important part of the internet, but it too suffered from abuse and manipulation for traffic.

For many DMOZ was the entry point to the web. Whatever you were looking for, you started there.

Google changed that, first for the better, then for the worse.

@ajsadauskas @degoogle I used to be one of those human editors. I was the editor of Scotland.org from about 1994 to about 1997, back in the days when it was exactly one of those hierarchical web directories – with the intention of indexing every website based in Scotland.

The tale of the internet has been curation, and I would describe it a little differently.

First we had hand made lists of website (Yahoo directory, or we had a list of websites literally written in pen in a notebook saying "yahoo.com" and "disney.com").

Then it was bot-assisted search engines like Google.

Then there was so much content we didn't even know where to start with Google, so we had web rings, then forums, then social media to recommend where to go. Then substack style email newsletters from your chosen taste makers are a half-step further curated from there.

If that is all getting spammed out of existence, I think the next step is an AI filter, you tell the AI what you like and it sifts through the garbage for you.

The reasons we moved past each step are still there, we can't go back, but we can fight fire with fire.

@ajsadauskas Back when, UW Madison hosted an outfit called The Internet Scout Project that was in the curation business for web resources. The decaying state of search (alternatively the growth of web resources intended to serve interests other than their visitors') has me thinking it would be good to work with public libraries to convene and host this sort of thing.

Librarianship is the right sort of ethos for it, and libraries are infrastructure for human-mediated discoverability.

@degoogle

@ajsadauskas @degoogle definitely something o be thinking about. More and more I’m using my followed hashtags, mastodon lists, and links to resources other people provide rather than just finding useful things in search results. But the big gap is still when I want to find quality info on a new topic. Cannot trust any of the damn results searching for how and how often to clean my kid’s new aquarium, for example. So much LLM and SEO crap info.

@ajsadauskas @degoogle

Yes to all. For a while I've been de facto using a miniscule subset of the web. My gateway to other, relevant websites are via human-to-human recommendations, primarily in a place like this.

This is how it's gonna go. we'll get human-curated search results, before someone "innovates" by mildly automating the process until someone "innovates" again by using AI to automate it further. Time is a circle

What's to say we won't have AI-curated lists and directories? That way we don't have to deal with link rot and the like. I think the issue is the algorithms used for search. We need better ones, better AI, not more frivolous human labor.

@ajsadauskas @degoogle
I've already seen new webrings forming.

Or maybe that was old webrings updating?

@ajsadauskas
I agree we need better and remember the early days well. Before indexes we passed URLs, in fact just IP addresses of servers we'd visit to see what was there, and that was often a directory of documents, papers etc. It filled us with awe, but let's not dial back that far!

Another improvement will be #LocalLLMs both for privacy and personalised settings. Much of the garbage now is in service of keeping us searching rather than finding what we want.
@degoogle

@ajsadauskas @degoogle i encountered https://aboutideasnow.com/ recently

it's not exactly what you describe and remember, but it's... at least real.

i was reminded by this post of my days from like 1998-2001ish on About.com (RIP)

And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.

True, but these things can also be used by us, to curate/maintain a high quality link collection. However, I'm not sure 'pages' will be read by humans in 5 years, so I have a feeling we wont need such a collection anymore. Well, not for humans but probably for our individual LLM's.

@ajsadauskas @degoogle hopefully they don't look like Dmoz, because i still have unpleasant flashbacks of that dark time 😋

@ajsadauskas @degoogle Curlie https://curlie.org/ is the continuation of the ODP

Just to add to your list of steps and consequences: I also think academic studies about information retrieval, indexing and crawling became less popular. Aspirant students hearing the message: those studies / workfields will become obsolete once AI does all that.

I remember a time when you could be a paper magazine every other week with curated lists of link on various topics. There were ads, but just paper ads :)

@ajsadauskas @degoogle a bit of history of Yahoo here, started as a web directory https://www.wired.com/1996/05/indexweb/

@ajsadauskas @degoogle Curation is elation.

@ajsadauskas @degoogle I mean we could still use all modern tools. I'm hosting a searxng manually and there is currently an ever growing block list for AI generated websites that I regularly import to keep up to date. You could also make it as allow list thing to have all websites blocked and allow websites gradually.