Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How might search engines work in a DAT enabled web?

(beaker's neat... i'll check it out)



I can't answer for pfraze, but I've thought about this a lot.

For one, even right now we badly need better client-side ways of caching, searching and indexing content. I should be able to easily cache, tag and search a wikipedia article, for example, without ever touching a remote server. On the decentralized web, these tools are critical.

As far as search goes, it'd be interesting to find a way of letting peers run anonymous queries on each others' local indices. Maybe you'd still only want trusted peers (e.g. friends of friends) to be able to query your index. Or maybe you'd want to have some privacy settings for who can search what content. Maybe this is a job for zero knowledge proofs? IDK. It's a hard problem for me to wrap my head around. How to balance privacy with fluid information retrieval.

This kind of p2p search system could be a powerful replacement of centralized search, especially in the sense that it gives users the opportunity to index content with humanly meaningful metadata.

Remember back in the day when people had web indexes? You'd visit someone's www page and they'd have a list of cool links? Think about a next level version of that, where peoples' curated indexes are stacked with meaningful metadata and well integrated with the browser.

I could also imagine interesting peer-graph indexing algorithms that worked using gather-apply-scatter kind of techniques.


macawfish is suggested distributed search indexes, which is a neat idea that YaCy [1] explores.

We're looking at adding a private search index in the browser itself. This index would be constructed by adding dat sites to your 'library,' and directing it to crawl the outbound links.

What's unique about this is, other than the private queries, we can establish trust relationships with all the data in your results. This is particularly useful for searching for users or software, since you want to trust those results. Dat is a cryptographic network - all sites, users, and apps are represented by public keys, and all data is signed - so your search results are a search against your personal Web of Trust.

When you search for "Paul Frazee," if you had previously added 'dat://beakerbrowser.com' to your library, then you'd see a result for "Paul Frazee (via beakerbrowser.com)". We can layer those signals too by matching URLs, so your result might be "Paul Frazee (via beakerbrowser.com, Alice, Bob)." Dat uses secure ledgers internally [2] so this is really a pretty decent form of key distribution, but of course it can be generalized to searching for all kinds of information. We just have to find out the limits of disk space usage.

I have more to say, but I'll add it with a reply to remain timely.

1. https://yacy.net/en/index.html

2. https://beakerbrowser.com/2017/06/19/cryptographically-secur...


Another interesting element about the local index is that it might be usable for discovery. Essentially there's a local cache of the Web in the user's browser. Much of that local cache has semi-structured data thanks to JSON, and we've begun to take advantage of that with a secondary indexer called WebDB [1]. It's feasible to imagine an API where apps ask the browser to search for files or Dat archives which match a certain spec. For instance, "Give me all user profiles which match the following JSON schema." Since the local index is curated by the user, this will naturally hit results that have some connection to the user.

The concept is of the browser as a user agent, crawling the Web from the user's input set to find relevant data, rather than an objective "map and rank the world objectively" engine like Google does.

Other than this local index and distributed indexes like YaCy (which are very neat but a lot of work) there's also the traditional hosted indexes. My hope is that people will be able to self-host their own indexes privately, as an extension of their subjective local index (but with a larger graph searched). However, there's still room for traditional search engines to crawl dat sites as well.

1. https://github.com/beakerbrowser/webdb




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: