Date   

Re: JDBC Batch

stephane
 

I mean when performing modifications, do something like:
 -1 set constraints deffered => disable foreign-key, unique constraints, ...
 -2 Batch updates table per table (addBatch, ...)
    - perform all Deletes from table A if any
    - perform all Inserts into table A if any
    - perform all Updates of table A if any

    - perform all Deletes from table B if any
    - perform all Inserts into table B if any
    - perform all Updates of table B if any

    - ...
-3 commit

(Let the constraints are checked at the commit time.)

--
Stephane


Re: JDBC Batch

Andy
 
Edited

You mean DELETE them in the datastore, do random operations, then re-CREATE them? Or do a datastore-specific disable (such as MySQL set foreign_key_checks=0). Nope.
But then its open source so anyone can contribute "features", not that I personally would be using that one you describe


Re: Problems with extra N+1 requests - likely due to poor mapping defns, but not sure how to rectify

Andy
 

That metadata extension works for me in this sample. How you apply it is up to you


JDBC Batch

stephane
 

Hi,

Are there any options to deactivate foreign-key constraints, then insert/update/delete table per table using batches, and then reactivate foreign-key at the commit ?
I used a product with that feature which was working at least on mysql and oracle.

thanks
--
Stephane


Re: How do the Query/Result caches and the L1 cache work together?

Andy
 
Edited

An L1 cache is an object cache, as defined by the JDO spec. You have an "id" you get the equivalent persistable object. It is used anywhere a persistable object is accessible, and that includes queries if the queries returns a persistable object. The log tells you where objects come from.

Query caching (in DataNucleus, not defined by the JDO spec) is a 3 level process ... generic compilation, datastore compilation, results. Compilations are always cached. Results are only cached when you tell it to cache them, via the query extension datanucleus.query.results.cached
It would make no sense to cache the results of every query for all types of software that use DataNucleus, hence it is off by default.


Re: Problems with extra N+1 requests - likely due to poor mapping defns, but not sure how to rectify

ebenzacar@...
 

I've tried to add the extension definition to my package jdo file but either it isn't working as I expect, or I've defined it incorrectly.  This is what I added:

<field name="addPerson" column="ADDPERSON" default-fetch-group="true">
<extension vendor-name="datanucleus" key="fetch-fk-only" value="true"/>
</field>

With this definition, I was expecting that the object retrieval would basically retrieve the FK value for addPerson from my Manufacturer object.  However, it seems to be trying to retrieve the full object, and puts itself in an endless loop for some reason, causing a StackOverflow.  The datamodel isn't exactly what I listed above, but rather has a bit of a circular design (ie: the AuditPerson actually contains an instance of Audit).   When trying to debug the issue further, I see that DN is calling `dnReplaceFields()` method on my Manufacturer object, but it is difficult to identify which field it is.   I can see the field index value which is causing the endless loop, but can't determine which fieldname the index maps to.  I cannot find the metadata/mapping information where the fieldname is mapped to the field number.  In my case, when I put the debugger on the dnReplaceField in the call stack, I see that it is trying to load field 56, and that this field is causing the infinite loop.  How/where can I identify which fieldname field 56 is mapped to?

Similarly, the DN logs show me what I think is a list of fields that are loaded/not loaded when the object is retreived from cache (loadedFlags) (ex: (id="zzzz" taken from Level 1 cache (loadedFlags="[YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYNNNNNNNNNNN]").  But how do I identify which index is which field?





I would have expected that setting datanucleus.maxFetchDepth=1 would prevent any recursion from happening, but it is still endlessly trying to reload that same field #56.

Removing the `addPerson/modPerson` from the default-fetch-group stops the infinite loop.

Any suggestions what/how to investigate next would be appreciated.

Thanks,

Eric




How do the Query/Result caches and the L1 cache work together?

ebenzacar@...
 

I've been encountering something I don't understand with the Query caches and the L1 cache.   I've tried reading the DN docs a few times, but I am still confused by the results I am seeing.

My persistence unit uses the default cache configuration, with no specific settings for the caches identified.  To my understanding then, the queryCache and the queryResult cache should be using soft references with unlimited size.  However, when I add profiling to my DB connection, I see the exact same query being executed multiple times.  For example, I will see the following query executed 4 times in a row:

exec sp_executesql N'SELECT ''izo.model.ActivePerson'' AS DN_TYPE FROM ACTIVEPERSON A0 WHERE A0.TYP = ''izo.model.ActivePerson'' AND A0.ID = @P0 UNION SELECT ''izo.model.Athlete'' AS DN_TYPE FROM ATHLETE A0 WHERE A0.TYP = ''izo.model.Athlete'' AND A0.ID = @P1 UNION SELECT ''izo.model.AthleteAgent'' AS DN_TYPE FROM ATHLETEAGENT A0 WHERE A0.TYP = ''izo.model.AthleteAgent'' AND A0.ID = @P2',N'@P0 bigint,@P1 bigint,@P2 bigint',4319,4319,4319

Normally, I would have expect this query to be added to the query cache, and the result to be in the queryResult cache.  Similarly, I would also expect the retrieved object(s) to be added to the L1 cache, and used for subsequent retrievals.

Questions:
1) Is the L1 cache only used if retrieving objects by Identity?
2) How does the query cache work?  What makes a query cacheable?  
3) Why would the query/result cache not be responding instead of re-querying the DB multiple times?


Thanks,

Eric


Re: Problems with extra N+1 requests - likely due to poor mapping defns, but not sure how to rectify

Andy
 
Edited

With a single object relation with the FK at the selected object side there are only 3 things that would be feasible.
  1. Load the related object, instantiating it. Put it in the fetch plan for this
  2. Don't load the related object and leave for lazy loading. JDO default.
  3. Include the PK (FK) field(s) in the SQL, and just store them some where and maybe use them later if required.
DataNucleus (and the JDO API also for that matter) allows options 1 and 2. Specification is explicit.

If you always want to load the related object with just the PK field(s) then you can always mark it as "fetch-fk-only" on that field, in the metadata.
If you only sometimes want to load the related object with just the PK fields, then put the field in the fetch plan for the SELECT, but also remove all other fields of the related object from the fetch plan (so it doesn't add a join to the related object).


Re: Problems with extra N+1 requests - likely due to poor mapping defns, but not sure how to rectify

ebenzacar@...
 

Thanks Andy.  Fair enough (embedded vs related).

I forgot to mention the fetch-groups.  Indeed, I have no fetch groups explicity defined - either in the mapping file nor in the API usage.  Everything is using the default fetch group.
But based on your explanation and rereading the DN docs for the N-teenth time, https://www.datanucleus.org/products/accessplatform_5_2/jdo/persistence.html#fetch_groups it clearly identifies that the default fetch group is essentially primitive+ values (ie: int, String, BigDecimal, Date, etc).  So I didn't have any expectation that the Related Objects will be loaded as part of the default fetch group (particularly with maxDepth =1), but was hoping/expecting that at least the FK references would be loaded at the same time to avoid the extra +1 calls just to retrieve the FK values.  

Reading the Kodo docs for the same background, I now see the difference.  In Kodo, they clearly indicate that the object will be left out of the fetch-group, but the FK will be loaded for it.  Whereas in DN, it seems that if the object isn't explicitly added to the fetch-group, the FK will be skipped as well.

I have now added the "addPerson" and "modPerson" to the default-fetch-group, and defined my maxDepth as 1.  And seems to work more inline with expectations.

Thanks,

 

Eric



I'm okay with the extra queries to retrieve the "addPerson" and "modPerson", but don't want the extra DB hit to retrieve their FK values independently.

Is there a way I can configure the defaultFetchGroup to "preload" the FKs during the initial object fetch, without identifying that I want the full objects loaded each time?

Thanks,

Eric

 


Re: Problems with extra N+1 requests - likely due to poor mapping defns, but not sure how to rectify

Andy
 
Edited

Audit fields "addPerson" and "modPerson" aren't embedded (exist in the same table), they are related (have their own table), using JDO terminology ... or at least without seeing your mapping (XML/annotations) that's what it looks like.

Your mapping definition controls where things are stored, and what is in the default fetch group. Your runtime API usage defines what is in the active fetch group.
What is in the fetch group defines what is fetched ... clue in the name. Kodo maybe didn't have JDO (2) fetch groups.

You rectify it by specifying the appropriate fetching (group) for the appropriate moment in the API usage.


Problems with extra N+1 requests - likely due to poor mapping defns, but not sure how to rectify

ebenzacar@...
 

I'm having some strange RDBMS N+1 calls most likely due to my mappings, but am not sure if this is a DN Persistence configuration issue or a mapping issue.  I'm using MSSQL.

My problem is that I have an Object "Person" which has some audit fields "addPerson" and "modPerson".  Both "addPerson" and "modPerson" are simple N-1 relations.  Conceptually like the following:

       //  InheritanceStrategy=New-table
Class Person extends Audit{
   String name;
   int age;
}


 //  InheritanceStrategy=subclass-table
Class Audit{
AuditPerson addPerson;
AuditPerson modPerson;
}


Class AuditPerson{
   String username;
}


When I retrieve the Person (using a datastore identity), I see an SQL Query which selects the primitive fields but not the embedded Objects.  Something like:
    "Select name, age FROM Person where ID = 123".

When I try to convert to a DTO and read the `addPerson`, and `modPerson` fields, DN then launches the next queries:
  • select addPerson FROM Person where ID =123
  • select username FROM AuditPerson where ID = <>

  • select modPerson FROM Person where ID =123
  • select username FROM AuditPerson where ID = <>



So, essentially, because of the "lazy loading", it needs to requery the DB for the 2 FK values in the Person table before it can actually query the AuditPerson table.


Clearly, I suspect that this is due to issues in my mapping definitions.  I am working with old mapping files from Kodo, so I suspect there is something I haven't done correctly.

Questions:
1) Is this normal/expected behaviour?  Is this due to the inheriteance model/hierarchy?  
2) Is this controllable via mapping configurations?
3) Is there a way to reconfigure the model such that I do not need the additonal independent queries to retrieve the FK values for the modPerson and addPerson.


I realize that this is difficult to diagnose without exact details and sample test case, but thought I would ask in the meantime while I try to reproduce the exact behaviour in a test app in case someone has seen/experienced this before.

Thanks,

Eric


Re: Any way to configure "development" mode for DataNucleus to force it to reload class metadata?

ebenzacar@...
 

Thanks for the tip.  I was looking for a `clearCache` or a `clearMetadata` method or something to that extent that I didn't notice the `unmanage/unload` methods.

I also found the plugin from Dan Haywood for Eclipse that he wrote a long time ago.  Am including the links here for future reference.
- https://www.danhaywood.com/2014/01/23/isis-integration-with-jrebel/
- https://github.com/danhaywood/isis-jrebel-plugin

I'm working with IntelliJ which has a different plugin architecture, and will have to look at this more carefully,  It will lead to many additional questions how to integrate DN with IJ more seamlessly, as currently the enhancement process must be run as a manual task each time a class is modified in IJ.  An IJ plugin that automatically triggers the enhancer for modified classes would be ideal.  

Thanks,

Eric


Re: Any way to configure "development" mode for DataNucleus to force it to reload class metadata?

Andy
 
Edited

There is a MetaDataManager for each PMF. It loads metadata when it needs to. There is a method unloadMetaDataForClass to unload a class' metadata if needing to do so.
There is also a method on JDOPersistenceManagerFactory unmanageClass(String className) which will call the underlying MetaDataManager method as well as the associated StoreManager method. Added in DN 4.0.

You could also ask on Apache ISIS support (which uses DataNucleus) because Dan Haywood played around with a JRebel addon to do something related some time ago. No idea if he published it.


Any way to configure "development" mode for DataNucleus to force it to reload class metadata?

ebenzacar@...
 

I'm working on a fairly large, slow startup legacy application in which I am slowly trying to integrate DataNucleus.  I am also leveraging JRebel, which allows me to modify most classes on the fly without having to restart the application everytime.  For most business oriented logic, this works great.

However, with the DN persistence classes, DN seem to load the metadata into a cache/singleton of some sorts, which prevents me from modifying/reloading a class on the fly, and forces me to stop the application server everytime and relaunch it.  I've sifted through the code and couldn't see anything to enable this, nor did I notice anything in the DataNucleus Metadata Properties so I'm trying to find if there is another mechanism I can use to clear the Metadata cache and have DN reload it on the fly.

Similarly, I would be happy with any process that allows me to modify the persistence class and re-enhance it without needing to restart the application everytime.

Thanks,

Eric


Re: Migration from Kodo: Any way to make it store foreign keys to later get already loaded objects from cache (for performance)?

ebenzacar@...
 

Hi,

I'm running into a similar problem as you where I need to create an ugly JDOHelper.getObjectIdAsLong() hack as well due to some legacy Kodo code.

Did you get something fully functional finally?  Can you share your plugin?  I've looked through the `org.datanucleus.store.rdbms.sql.method.JDOHelperGetObjectIdMethod` implementation, but frankly a little lost by the way it processes everything.  I'm a little unclear on the use of a NumericExpression vs an ObjectLiteral, etc.

Thanks,

Eric

 


Multitenancy query result error - query cache key (bug ?)

stephane
 

Hi,

When 2 queries with the same tenant but different tenantReaders are executed, the second one returns the result of the first one.

The SQL generated is the same for the second query while it should be different.

The query cache key only uses the tenant which may explain the case.
String multiTenancyId = ec.getNucleusContext().getMultiTenancyId(ec);
if (multiTenancyId != null)
{
queryCacheKey += (" " + multiTenancyId);
}
(no test case yet)

--
Stephane


Re: Strange behaviour running in WIldfly 10 with EmulatedXAResource

Andy
 
Edited

You are using a JEE connection pool but are setting the resource type as "local" hence DN will try to commit the connection (since it commits local things). Your connection (JBoss JCA) then throws an exception because it wants to commit things, maybe?. Use a local connection if using local ? But then I don't use JEE containers ...

You also have "datanucleus.connectionPoolingType" yet have already specified "connectionFactoryName" to be from JBoss. Either DataNucleus creates the connections to the datastore or JBoss does, but you can't have both, clearly.


Strange behaviour running in WIldfly 10 with EmulatedXAResource

ebenzacar@...
 

I'm encountering a strange behaviour for which I have not been able to pinpoint the cause and was hoping that someone could help point me in the right direction.

I have an application running under  JBoss EAP 7.0 (Wildfly 10), but only having the datasource specified at the container level.  I am trying to fully manage the JDOPersistenceFactory from within the JEE (EJB) application.

My persistence.xml is configured as follows (located in my EJB jar in the META-INF/ folder:
<persistence-unit name="cache" transaction-type="RESOURCE_LOCAL">
<non-jta-data-source>java:jboss/datasources/DS</non-jta-data-source>
<mapping-file>package.jdo</mapping-file>
<exclude-unlisted-classes/>
<properties>
<property name="datanucleus.connection.resourceType" value="RESOURCE_LOCAL"/>
<property name="datanucleus.transaction.type" value="RESOURCE_LOCAL"/>
<property name="datanucleus.storeManagerType" value="rdbms"/>
<property name="datanucleus.schema.autoCreateAll" value="false"/>
<property name="datanucleus.ConnectionFactoryName" value="java:jboss/datasources/DS"/>

<property name="datanucleus.connectionPoolingType" value="dbcp2-builtin"/>
</properties>
</persistence-unit>

I have instantiated my connection factory statically as follows:
public class DataNucleusPersistenceManagerFactory {

// hack to provide access to the pmfs from a static context
private static Map<PersistenceUnit, PersistenceManagerFactory> pmf = new ConcurrentHashMap<>();

static{
PersistenceManagerFactory persistenceManagerFactory = JDOHelper.getPersistenceManagerFactory("cache");
pmf.put(PersistenceUnit.CACHE, persistenceManagerFactory);
}

/**
* Retrieve PersistenceManagerFactory from legacy code
* @param type
* @return
*/
public static PersistenceManagerFactory getPersistenceManagerFactory( PersistenceUnit type){
return pmf.get(type);
}


All my Stateless EJBs are defined as BeanManagedTransactions by extending a base class with the following interceptor defined:
@TransactionManagement(TransactionManagementType.BEAN)
public class BaseManagerBean implements BaseManager {

PersistenceManager pm;

@AroundInvoke
public Object log(InvocationContext inv) throws Exception {
pm =
DataNucleusPersistenceManagerFactory.getPersistenceManagerFactory( CACHE );
pm.begin();

Object ret = inv.proceed();
pm.commit();

}

}

This allows for the logic for my bean method to simply use the pm within its logic without needing to worry about transaction boundaries/etc.  I have simplified the interceptor for illustration purposes.

My issue that I am encountering is when I have nested EJBs (ie: one EJB which calls another).  In this context, I end up with nested transactions, since each EJB method call will create a new PM and manage its own transaction.

When this happens, the inner transaction completes successfully, but the outer one throws and exception that:

Caused by: java.sql.SQLException: IJ031019: You cannot commit during a managed transaction
        at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.jdbcCommit(BaseWrapperManagedConnection.java:1063)
        at org.jboss.jca.adapters.jdbc.WrappedConnection.commit(WrappedConnection.java:834)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource.commit(ConnectionFactoryImpl.java:734)
        at org.datanucleus.transaction.ResourcedTransaction.commit(ResourcedTransaction.java:348)
        at org.datanucleus.transaction.ResourcedTransactionManager.commit(ResourcedTransactionManager.java:64)
        at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:417)
        at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:288)
        at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:99)
        ... 128 more
 


Stepping through the code, falling into the IronJacamar libraries, I see that the ConnectionWrappers are treating this as an XA transaction.  I also see that from the `org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource` resource which the DN CF creates.

This is where I get confused.  I'm not sure why this is happening, nor if this is "normal"/expected behaviour.  I have tried to reproduce this in a small test application by embedding 2 txs, with the same PU defined, but it works properly there.  I have not yet tried embedding two EJBs in a test application.

I can only suspect that I have an errant configuration somewhere which is causing this, but cannot identify what.  My Datasource is configured the same for my test application.  Just for reference:

<datasource jndi-name="java:jboss/datasources/DS" pool-name="DS" enabled="true">
<
connection-url>jdbc:sqlserver://${db.tx.host};databaseName=${adams.db.tx.name}</connection-url>
<
driver>sqlserver</driver>
<
pool>
<
min-pool-size>5</min-pool-size>
<
max-pool-size>2000</max-pool-size>
</
pool>
<
security>
<
user-name>${adams.db.tx.username}</user-name>
<
password>${adams.db.tx.password}</password>
</
security>
<
validation>
<
check-valid-connection-sql>select getdate()</check-valid-connection-sql>
</
validation>
</
datasource>
<drivers> <driver name="sqlserver" module="com.microsoft.sqlserver"> <xa-datasource-class>com.microsoft.sqlserver.jdbc.SQLServerDriver</xa-datasource-class> </driver> </drivers>


What am I missing, doing wrong?  Do I have a incorrect or missing configuration?  What is  DN / WF seeing this as an XA transaction?  Or is this expected behaviour when working with nested EJBs due to the inherent stateless / pooling behaviour of EJBs?  If the latter, am I forced to move to Container Managed Transactions such that no bean commits its own transaction?  If not, how would I know which transaction is the outer transaction?

Thanks for any insights.

Eric


Re: How to configured the PersistenceManager for Bean Managed Transactions?

Andy
 

https://www.datanucleus.org/products/accessplatform_5_2/jdo/persistence.html#jboss7 says that "DataNucleus JCA adapter supports both Local and XA transaction types".


Re: Proper configuration with the JCA connector in WIldfly

Andy
 
Edited

FYI The JCA was written by people who no longer develop DataNucleus, and is dependent on volunteers. Consequently you're pretty much on your own. Clearly JDO is just as usable outside of JCA, creating a PMF and PM's as required.

101 - 120 of 446