What does HackerNews think of magnetico?

From https://github.com/bitmagnet-io/bitmagnet/issues/11

> The DHT implementation was largely borrowed from Magnetico (https://github.com/boramalper/magnetico) a popular and widely used app which is likewise unable to service these requests.

You can't build a decentralized search because there's no way to trust whatever results you get until you actually build it. If you don't want to rely on a specific community, your best bet is to crawl yourself and search locally: you can do that with magnetico (https://github.com/boramalper/magnetico/). Don't be frightened by the fact that it is archived, it works.

The problem then will be, how do you make sure your content is legit? There's no magic way here, the best thing you can do is compare the number of seeders and aim for the highest. If a torrent is fake, people will delete it and it won't be seeded. I have a thingy for that: https://sr.ht/~rakoo/magneticos/

The problem then becomes, number of seeders naturally selects towards popular content. It doesn't ensure viability of content. But I don't think there's a technical answer to that.

I just get a rickroll gif on your link. Did you mean this project?

"Autonomous (self-hosted) BitTorrent DHT search engine suite."

https://github.com/boramalper/magnetico

> you can passively collect info from the DHT swarm and build up an index over time

For anyone wanting to pursue this, I feel like I can share, I used this recently https://github.com/boramalper/magnetico, people share database dumps regularly. I found ~4 DB dumps and merging scripts are all you need to get up and running.

Magnetico[1] is an easy-to-use, self-hosted DHT crawler. It has a web UI that shows statistics and gives you a search engine (it uses sqlite full-text search).

It needs quite a bit of bandwidth and some storage space, but worked well so far, for me. I've been running it for a couple of years and it indexed around 12.5M torrents (36GB of uncompressed database).

[1]: https://github.com/boramalper/magnetico

I'm running magnetico (https://github.com/boramalper/magnetico) on my VPS. I currently have an index of 1.6M magnet links stored in a 5GB database.

SQLite is most interesting not when the database is small, but when there are very few writes and all you do is reading. You can also look at https://datasette.io/ and see how SQLite is perfect for representing a lot of datasets and querying them

I'm using magnetico (https://github.com/boramalper/magnetico), a selfhosted bittorrent DHT scraper. After a few months of running I now have a little bit under 1M magnets in a ~3G database. SQLite is useful not just because I can query it with the full power of a relational database, but because I'm using its FTS capability for searching.

If I had to install a typical database or some search engine I would never have used it. It is more than enough for what I'm using it for.

Having a distributed search engine is far from easy (there's the main issue of trust in the index). It's possible to do it the other way around, thanks to DHT scraping being allowed: everyone can scrape the peers and build an index at home, with tools like magnetico (https://github.com/boramalper/magnetico) for example

It's hard to imagine how things could be different, but for many services you don't need a middleman. For a long time, banking was thought to be a necessary middleman because you "have to" trust at least some actor to verify the transactions. Turns out you don't need to, with distributed ledgers (Bitcoin, etc).

When it comes to search engines, two things shake the centralized assumption: (1) you can crawl DHTs like https://github.com/boramalper/magnetico does for BitTorrent, and this can be done self-hosted, (2) too often we assume that searches must be global, but in many cases you are implicitly constraining your search space to regional boundaries or interest boundaries. E.g. as a westerner you are probably not interested in content from Vietnam or Pakistan. Also, when searching the Web, rarely do people browse through multiple Google search result pages. As a consequence, things like `awesome-*` lists ( https://github.com/bayandin/awesome-awesomeness ) are a good starting point for a decentralized curated catalogue that could cover that use case. So you could imagine a curated catalogue of programming websites compiled as a few hundred megabyte file, it could pretty much cover your interest boundaries, and as a file this can be shared over Dat today very easily. Want to search another 'interest boundary'? Run the same search software but on a different catalogue file. These things are possible.

In other words, centralization is overestimated. Maybe there are a couple of genuine centralized-middleman use cases but most of use cases can be very well covered by decentralization.

I think what he's more talking about is something more akin to a distributed search system than a blockchain - you could make the magnet links last forever in a blockchain, but that doesn't really make sense if no one holds the files anymore you're just keeping useless links around and storing them on every single node for no reason.

A search DHT makes much more sense, look up keywords based on hash. Think the Kademlia network from the ED2K days. Maybe I'm being too academic by even suggesting a DHT be used, you could do it in simpler ways like flood search in the style of Gnutella - any distributed search means would work really.

I believe there are a few attempts at this in the bittorrent world, like Tribler, which is probably the most practical implementation to date, there's a few others too, none of which look particularly mature yet:

https://sourceforge.net/projects/aresgalaxy/editorial/?sourc... https://github.com/lmatteis/torrent-net https://github.com/boramalper/magnetico

The problem with distributed search is that perfecting it is hard - bittorrent has won out because websites could be used to prevent spamming of malware, track reputation and discussion of torrents and individuals, etc. Tribler has some proposed alternatives here and I seem to remember Kad having done a decent job preventing this from becoming a significant problem, but it never had the popularity.

If someone manages to knock out all the big torrent and usenet indexes overnight, these systems will become a lot more necessary and probably get a lot more popular.