There is a minor downside to RSS limiting: Because RSS memory allocations are on-demand and asynchronous with tasks, applications can’t intercept them, requiring the kernel to kill the application when limits are exceeded. VSS limiting avoids this by having the kernel return an error when a limit-exceeding memory request is made, allowing the application to handle the error.
VSS limiting, however, has a much more significant problem: Applications’ memory requests are grossly inaccurate. They routinely exaggerate their memory needs. For example, a Java virtual machine (JVM) can allocate 4GB of virtual memory and then use only 1MB. This can leave VSS limits with little practical meaning.
In summary, RSS limiting is generally a far more effective limiting practice than VSS limiting.