I use PrePersist, PreUpdate, ... callbacks to perform validation on object graphs. When an error is identified, the callback raise an exception ( instance of a sub-sub-class of RuntimeException).
I expect, the transaction set as rollback only in such cases.

As DN5.2.0 does not set the transaction as rollbackonly, the tx can be committed (and I do it ...)
I checked the JPA Specification to be more sure:

Lifecycle callback methods may throw unchecked/runtime exceptions. A runtime exception thrown by a callback method that executes within a transaction causes that transaction to be marked for rollback if the persistence context is joined to the transaction

I tried datanucleus.jpa.txnMarkForRollbackOnException=true but without success.

So I looked at DN code. Transactions are marked as rollback by JPAEntityManager.throwException. This method seems only called on exception, if the exception raised is an instance of NucleusException.
ex in persist method:

catch (NucleusException ne)
{
    throwException(NucleusJPAHelper.getJPAExceptionForNucleusException(ne));
}

Could it be safe, correct, possible ... to also catch RuntimeException like

catch (RuntimeException ne)
{
throwException(NucleusJPAHelper.getJPAExceptionForNucleusException(ne));
}

Better option (to isolate this particular situation from all other possible causes of RuntimeException being thrown during persist) would be to trap any RuntimeException from user callbacks, and rethrow as a NucleusUserCallbackException (or some name like that), with the RuntimeException nested inside it. Then "throwException" can check for NucleusUserCallbackException.

You can always provide a Pull Request.