Subject Individual security2.fdb per tcpsvd


We have quite a special fb setup for a number of customers sharing a server but trying to separate out the resources and allow individual fail over. On day 0 we intended to use containers and VMs but came to the conclusion that they were themselves more resource heavy and complicated than the solution we came up with which was:

per customer ip address / DNS name

per customer partition using drbd to synchronise to a separate box

single nginx listener spawing per customer phpfpm based on DNS

per customer tcpsvd /usr/sbin/fb_inet_server bound to ip address with lock files in a shared tmpfs

This has given us quite a nice division of resources on the server with the ability to swap customers between servers without affecting each other, and able to turn down customers at will and limit their resources.

We've just had a nightmare couple of weeks having tripped over where neither us nor our hosting provider could see any idea where the load or locking or blocking was happening. This had the effect of bringing down all customers together whenever one of them triggered the sweep, and generally ground the server connection times into the floor over time - I have a lovely graph of connection times from the last month which just caused me despair at the time I made it and saw everything getting worse and worse!

This has focused my attention on sercurity2.fdb and I've come to the conclusion, maybe wrongly, that we should have a separate one per customer. At the moment all customers are hitting the same file, and adding to the same lock file - but everything else they do hits their own files, filesystems, and lock file.  At the moment I don't have any stats to prove this concern (I'm still breathing deeply after CORE-5067) but the two questions I've got are:

1. Should I be worried about / would it give me any performance benefit to have separate security2.fdb per tcpsvd/customer ?

2. How do I set that up?

Obviously the 2nd one depends on the 1st, and the 1st only matters if the 2nd is possible! 

We have about 25 customers per server, each with databases ranging from 1GB to 12GB. tcpsvd is set to spawn a maximum of 60 connections per customer at any one time, and a typical customer generates about 80 to 120 connections a minute.