> What can be the problem? How can I find and solve it?

This is definitely resources leak. May be caused by virtually
anything in your system. AFAIU you have application always connected to
server, so to such usual source of leaks as bad UDF, we may add not
freeing such connection-long resources as statement handles and,
probably, temporary BLOBs.
Check if memory consumption return to normal if you disconnect all
clients from server.

SY, SD.