One of the interesting features of Elliptic Curve Cryptography is that the computation time (including the required entropy) is roughly linear to the key size used. This means that using P521 will take roughly double the time that P384 will.
I assume that you are choosing these very strong curves due to the importance of your connections, however there is a tradeoff of performance for security.
A simple test would be to remove the curvePreferences, and allow the system to auto-negotiate a cipher suite. You could also set an AES-256 cipher suite such as TLS_RSA_WITH_AES_256_GCM_SHA384, and see how the system performs. This should give you an idea of how TLS would work under normal situations.
If using auto-negotiated suites performs well, then you will have to decide if the cost of higher crypto is worth it to you.