@sir i don't think it's a problem. the web is large now and webservers need to account for that, unless there's some specific bug that's causing things to get hit more than there are eyeballs
@sneak every time you post a link on mastodon, every instance it gets federated to will automatically fetch it to generate a link preview
@sir @sneak what would be the solution?
A centralised proxy? Previews fetched from their origin (and DDossing that)? Some distribution and mirrorring; and then how to avoid poisoning?
Honestly curious.
What about a separate p2p 'linkpreview' network? Such a project could even leverage some 'reward' system, not unlike filecoin, dat or storj. Mastodon, matrix, signal etc could then fetch previews there, instead of all implementing their own.
@berkes that's 40r/s. any website that can't handle 40r/s shouldn't be on the internet
@sneak I've owned and/or founded three webhosting companies. Please believe me when I say that your requirement will exclude over half of the "long tail" of the internet. If not more. Too many websites are on a constant brink of breakdown. From webshops that can handle 5 orders a week, but never more to blogs that crash at anything over 1rps.
I cherish that web. I certainly don't want to "exclude it" from being shared on fediverse. Not now, and not when the fediverse grows 4, 10, or 100x.
@berkes it won't exclude it from being shared on the fediverse. i would be extremely surprised if the links don't get posted even if preview generation fails. there is no problem here
@berkes also there isn't a requirement. you can post any link you like, including ones that *can't* handle 2k hits per minute. it'll just be down/slow for a few minutes until the link preview fetchers time out. there is no problem here.
@sneak there's a difference between a. "sharing a link on twitter" that will cause 1 server to fetch a preview, and b."the fediverse" that causes 2.4k servers to fetch a preview *within a minute*.
And c. N visitors over the next hours.
More important, IMO is the message you want to send: do we encourage decentralisation? selfhosting, many small nodes, etc.?
Or do we say "if you post a link to the fediverse, there is a requirement that your server can handle 2k hits within a minute."