On 11-8-2018 12:33, 'River~~' river14april@... [firebird-support]
wrote:

> I suggested wget as it has options to cope with that issue.  I recommend
> these:

[..]

> The advantage of taking a precautionary backup is that there is
> (probably!) no need to hurry. Take it slow and you won't make take too
> much bandwidth from other Yahoo customers, won't make Yahoo's problems
> worse, and won't fall foul of their rate limiter.
>
> See
>
> https://www.gnu.org/software/wget/manual/wget.html#Recursive-Download
>
> to figure out which options you need to define your recursive download.
> You can avoid picking up graphic files for example.

I used the Yahoo Groups REST api instead, which is allows for requesting
the messages by a simple id (which is a non-random incrementing id), no
need to crawl or perform recursive downloads, and it allows for download
resume, and incremental updates when new messages have been added.

I wrote something in Java to do it for me, and either Yahoo doesn't do
rate limiting on that API, or I now have a corrupt archive ;) (checks of
random messages haven't revealed issues though). In any case, I managed
to archive firebird-support in a few hours.

I will play around to see if I can do something fun with that.

Mark
--
Mark Rotteveel