Subject Re: [Firebird-Java] Are there plans for streaming backup support?
Author Ivan Arabadzhiev
Post date 2016-02-22T10:50:37Z
Hi again,
1 mark()/reset() is why I switched to BufferedInputStream to begin with and it appears to be holding as a solution. I went through the GNU implementation and it appears to be simple enough to not add any significant overhead. I could check for markSupported() but I'd rather have a specific stream on top (perhaps add a check to not wrap when given BufferedInputStream), mostly because restore is ... horrific :) and that way I can rely on some consistency

2 I did look at fbsvcmgr.cpp and noticed it passes isc_info_svc_timeout, which I couldn't exactly figure out how to add (byte sequence is [64,4,0,1] and I got lost in Jaybird's way of figuring out what param is what). Fortunately, it appears not to matter as my last commit at 2 am seems to be the final piece (more explanation coming in a few lines)

3 I live in Emacs/Chrome and I don't really do raw Java that often (read 'haven't tuned it that much') so at PR time I will once again pass things through the Eclipse formater, which I will tune however you wish (if you want, you could pass me a settings file to use) - last time I made a new workspace specifically for jaybird. IntellyJ is also not a problem

4 After the commit last night I left my machine restoring the problematic case while(true) and woke up to it still going, so I'm thinking of extracting FBBackupManagerBase, formatting squashing and making a PR so we can continue the discussion there?

Now ... for the restore :
Long version (feel free to skip) goes like this - I've been tech support since I was ~13 years old so backups have caused me a lot of relief (and pain over the years) and I did jump at StreamingBackups when they were introduced. In fact, I'm the one mentioned in the first .Net provider release to support them :) So when I finally started on a Jaybird, even before Jiri suggested, I figured I'd take a look at an implementation I've personally tested. Now, it would appear I haven't used it enough to hit the bug I described last time, because the code starts with adding the request codes and skips isc_info_svc_line if the backup isn't verbose (which fbsvcmgr.cpp doesn't do). For whatever reason, the FirebirdSQL engine would decide from time to time that it has had enough input if isc_info_svc_line isn't present in the request. Didn't find a pattern but no amount of restarting or deleting the .fdb file helps and some combinations of fbk+engine would not restore more than 2-3 times in a row.
--------------------------------------------------------------------
Short version :
isc_info_svc_line needs to be present in the SRB even on non-verbose backups or not even god knows if restore would ever finish. Thought of reporting it to firebird-devel but it appears to be the way fbsvcmgr works so that might be the actually expected behavior. It also appears to help with my concerns about receiving a lot of 0-length stdin responses and drops CPU load dramatically (both on the JVM and the FirebirdSQL side). The effect is not *that* noticeable on your typical server but one of my test machines is a Cubieboard2 which experiences about half the load after the last commit :)

2016-02-22 12:15 GMT+02:00 Mark Rotteveel mark@... [Firebird-Java] <Firebird-Java@yahoogroups.com>:
 

<1455738179.3877554.524148514.7D001732@...> <CAGiAgVe-vEpn44zwYoaOkZP8XdriQeKWvHgQDLshaLh63kkG+A@...> <CAGiAgVcxja1QpjcYHz-Rf+fjyDYhbMLXiq5q4_akYoe7QSE+uw@...> <CAGiAgVdoVrJqET5xrFDkQGYhQLhkqW3R2cESeR95MN5r0y=Sjg@...>
Message-ID: <c4b5bb69acdafd107020994e6bea6583@...>
X-Sender: mark@...
User-Agent: RoundCube Webmail/0.2

A quick note: I wasn't able to finish reviewing your code this weekend,
and I am afraid I - probably - don't have time nor energy until next
weekend.

I noticed that you used `mark` (and `reset`) to check if you already
reached end-of-stream, unfortunately not all streams support marking. This
needs to be addressed in some way, for example check `markSupported`, and
if false wrap in a `BufferedInputStream` or instead of using `mark`, we
could use a `PusbackInputStream` (so the read can be 'pushed back'), or
maybe we need to do this differently (eg reading farther ahead).

I went digging in the Firebird code to see how the remote backup/restore
worked, and I got a bit lost. I also looked at the .net provider and I am
considering whether we should divide this in two classes as well (one for
backup and one for restore), just like is done in the .net provider.

If you haven't yet done so, use the formatting definition in
build/formatting, as I noticed your code is using tabs instead of 4 spaces.
If you already did, let me know which one you used (intellij or eclipse),
as it could mean something is wrong in the formatting definition.

Mark

On Sun, 21 Feb 2016 22:07:03 +0200, "Ivan Arabadzhiev intelrullz@...
[Firebird-Java]" <Firebird-Java@yahoogroups.com> wrote:
> Hi again,
> Code is quite stable but I did encounter a glitch while testing - I have
a
> .fbk, created by x64 Windows Firebird, which will throw my restore in an
> endless loop with a linux server (restores fine to Windows). I do see a
> difference in the fbk, depending on which server created it and in that
> specific case the server stops requesting more info 2 bytes before EOF.
> Just to make things even more fun, every now and again it manages to
> restore ... which I find a bit strange.
> The good news is every server I tested can restore its own backups (with
> any JVM on the client side).
> The other good news is that fbsvcmgr doesn't have that problem so I'm
> looking into how things are handled over there.

 Was it a transportable or non-transportable backup?

Mark