Broken links to the previous mailing list archive

Hi,

lists.linuxfoundation.org is no longer hosting the static HTML email archives for bitcoin-dev or any other mailing list. We knew that this might happen and prepared months ago with the mailing list migration and also with various backups: https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/

However, many thousands of links around the web are still pointing to the old archive. This is really unfortunate as there is a wealth of content in the archive and these links were considered canonical. This is a general plea to update any broken links that you might be able to fix.

What you can do, options ========================

There are a few options for actions that effected content providers can take:

1) Use `gnusha.org/url`. Explained below.

2) Replace lists.linuxfoundation.org urls with resolved urls to a new location for mailing list archives. More on this below.

3) Do nothing. Wait and see. It is possible that we can plead our case to Linux Foundation and get them to host static archives indefinitely, or at least host redirects.

4) Replace broken links with Wayback Machine urls. Someone told me apparently their archive is incomplete? I don't recommend this one because I am concerned about stress and strain on the Internet Archive Wayback Machine service beyond the additional activity that these links would incur. Besides, Wayback is already a highly centralized point for internet archival activity and therefore some decenteralization is warranted here.

gnusha.org/url redirect service ===============================

This is a redirect service I made that takes a URL parameter and redirects. To use it, get a previous url like:

`https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html`

your new url will be:

`https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html`

and it will redirect to a current mailing list archive location for that message:

`https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/`

It uses a mapping file based on the mailing list archives. I have published scripts to make this mapping here: https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b

See https://gnusha.org/url for more information. Thank you to Jonas Schnelli for suggesting the creation of this redirect service.

Manual resolution =================

How to "manually" resolve "without" the gnusha.org/url redirect service. Use resolver.py here: https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py and set RESOLVER_HOST to `https://gnusha.org/url`, also set `resolver = resolve_locally` to `resolver = resolve_remotely`.

This will convert urls from the old LF format to resolved urls like: `https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/` or `https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/` depending on which public-inbox mirror you choose to use.

Why resolve or not ==================

I'll elaborate on `gnusha.org/url` vs `gnusha.org/pi/bitcoindev` style urls.

It is nice and simple to use a find-and-replace with `gnusha.org/url/` because anyone can easily repeat that exercise and verify the git diff before merging the pull request. However, the actual routing is not something that is verified by just checking that the `git diff` looks sane.

If I was to provide my mapping (I haven't uploaded this yet, but I did publish the scripts I used to make it, hopefully someone will re-generate it to verify my work), then someone could write a script and/or cross-verify that all of the replacements are valid according to the mapping, if we were to use the destination urls instead of the redirect service for this pull request. Then someone would be able to verify that kind of pull request.

There are other further issues though, like content malleability, wrong redirect link injection, service provider compromise... instead of referencing by Message-ID maybe in the future we can reference by email hash.... But now I'm dreaming.

- Bryan https://x.com/kanzure

I have two points.

(1) I think it is a good idea for the Bitcoin mail lists to purchase a domain of its own in the future because it is hard to expect any open-source foundation to long live Bitcoin.

(2) From the previous mail https://gnusha.org/pi/bitcoindev/CABaSBaz9OTSVa1KNk0GOrH3T-kRF_7OPVu0AtpuaFGVB=zhdwQ@mail.gmail.com/, it seems that Linux Foundation may be happy to do some redirects, maybe it could be coordinated?

"Linux Foundation has either offered or agreed to maintain archive permalinks so that content of historic importance is not lost. Fortunately for us while lists.linuxfoundation.org mailman will go down, they have agreed the read-only pipermail archives will remain online."

Weikeng On Wednesday, November 13, 2024 at 2:55:58 AM UTC+7 Bryan Bishop wrote:

> Hi,
>
> lists.linuxfoundation.org is no longer hosting the static HTML email
> archives for bitcoin-dev or any other mailing list. We knew that this
> might happen and prepared months ago with the mailing list migration
> and also with various backups:
>
> https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_Z...@mail.gmail.com/
> <https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/>
>
> However, many thousands of links around the web are still pointing to
> the old archive. This is really unfortunate as there is a wealth of
> content in the archive and these links were considered canonical. This
> is a general plea to update any broken links that you might be able to
> fix.
>
> What you can do, options
> ========================
>
> There are a few options for actions that effected content providers can
> take:
>
> 1) Use `gnusha.org/url` <http://gnusha.org/url>. Explained below.
>
> 2) Replace lists.linuxfoundation.org urls with resolved urls to a new
> location for mailing list archives. More on this below.
>
> 3) Do nothing. Wait and see. It is possible that we can plead our case
> to Linux Foundation and get them to host static archives indefinitely,
> or at least host redirects.
>
> 4) Replace broken links with Wayback Machine urls. Someone told me
> apparently their archive is incomplete? I don't recommend this one
> because I am concerned about stress and strain on the Internet Archive
> Wayback Machine service beyond the additional activity that these
> links would incur. Besides, Wayback is already a highly centralized
> point for internet archival activity and therefore some
> decenteralization is warranted here.
>
>
> gnusha.org/url redirect service
> ===============================
>
> This is a redirect service I made that takes a URL parameter and
> redirects. To use it, get a previous url like:
>
> `
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html`
> <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html>
>
> your new url will be:
>
> `
> https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html`
> <https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html>
>
> and it will redirect to a current mailing list archive location for
> that message:
>
> `https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1...@blaine.gmane.org/`
> <https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/>
>
> It uses a mapping file based on the mailing list archives. I have
> published scripts to make this mapping here:
> https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b
>
> See https://gnusha.org/url for more information. Thank you to Jonas
> Schnelli for suggesting the creation of this redirect service.
>
>
> Manual resolution
> =================
>
> How to "manually" resolve "without" the gnusha.org/url redirect
> service. Use resolver.py here:
>
> https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py
> and set RESOLVER_HOST to `https://gnusha.org/url` <https://gnusha.org/url>,
> also set `resolver
> = resolve_locally` to `resolver = resolve_remotely`.
>
> This will convert urls from the old LF format to resolved urls like:
> `
> https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_Z...@mail.gmail.com/`
> <https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/>
> or
> `
> https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_Z...@mail.gmail.com/`
> <https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/>
> depending on which public-inbox mirror you choose to use.
>
>
> Why resolve or not
> ==================
>
> I'll elaborate on `gnusha.org/url` <http://gnusha.org/url> vs `
> gnusha.org/pi/bitcoindev` <http://gnusha.org/pi/bitcoindev> style urls.
>
> It is nice and simple to use a find-and-replace with `gnusha.org/url/`
> <http://gnusha.org/url/>
> because anyone can easily repeat that exercise and verify the git diff
> before merging the pull request. However, the actual routing is not
> something that is verified by just checking that the `git diff` looks
> sane.
>
> If I was to provide my mapping (I haven't uploaded this yet, but I did
> publish the scripts I used to make it, hopefully someone will
> re-generate it to verify my work), then someone could write a script
> and/or cross-verify that all of the replacements are valid according
> to the mapping, if we were to use the destination urls instead of the
> redirect service for this pull request. Then someone would be able to
> verify that kind of pull request.
>
> There are other further issues though, like content malleability,
> wrong redirect link injection, service provider compromise... instead
> of referencing by Message-ID maybe in the future we can reference by
> email hash.... But now I'm dreaming.
>
>
>
> - Bryan
> https://x.com/kanzure
>

On Tue, Nov 12, 2024 at 06:35:31PM -0800, Weikeng Chen wrote:

>
> (1) I think it is a good idea for the Bitcoin mail lists to purchase a
> domain of its own in the future because it is hard to expect any
> open-source foundation to long live Bitcoin.
>

Unfortunately there is no "Bitcoin mailing list" entity that could purchase a domain or hosting on its own. It would be possible to set one up but ultimately it'd ground out in community members volunteering their own resources to maintain the archive. (This is exactly what is happening with the new gnusha.org archives.)

I suppose in theory somebody with a lot of cash could set up an org with a charter and an endowment that could pay some sysadmin forever...but it's hard to see anybody stepping up for the sake of email archives. And as we've seen from our own problems with the Linux Foundation servers, and from the recent hack on the Internet Archive, that even this isn't likely to be sufficient.

The hope by using the LF servers was that, because they are a large and diversified organization, and ideologically aligned with providing access to information, that they would stick around. It's pretty frustrating and disappointing that this isn't the case.