> "C:\Program Files\Firebird\Firebird_3_0\gbak.exe" -b -v -t -se service_mgr employee e:\xxx.bak -user SYSDBA

> works if no one is connected else i always get error "file is used by another process".

  Specify remote (network) access explicitly (else it tries to open database using embedded engine and fails):

gbak.exe -b -v -t -se xnet://service_mgr employee e:\xxx.bak -user SYSDBA

or

gbak.exe -b -v -t -se localhost:service_mgr employee e:\xxx.bak -user SYSDBA