Thanks; will look into that more closely.

Can you give me a pointer where to find the fieldindex -> fieldname mapping?  I was am unable to identify for a certainty which index is which field.   I'm not sure where to look for it.

Thanks,

Eric

opensource development on such complex product (closely comparable to database software development) requires a big big expertise on the product design, constraints, ... which I don't have (yet) on DN.

Step by step ...
--
Stephane

JDO/JPA API calls are processed in the appropriate order. If a required statement is batchable then it is batched. Referential integrity is always assumed.
https://www.datanucleus.org/products/accessplatform_5_2/datastores/datastores.html#statement_batching

There is no mode to do "all table X followed by all table Y followed by all table Z". Open source ...

I mean when performing modifications, do something like:
 -1 set constraints deffered => disable foreign-key, unique constraints, ...
 -2 Batch updates table per table (addBatch, ...)
    - perform all Deletes from table A if any
    - perform all Inserts into table A if any
    - perform all Updates of table A if any

    - perform all Deletes from table B if any
    - perform all Inserts into table B if any
    - perform all Updates of table B if any

    - ...
-3 commit

(Let the constraints are checked at the commit time.)

--
Stephane

You mean DELETE them in the datastore, do random operations, then re-CREATE them? Or do a datastore-specific disable (such as MySQL set foreign_key_checks=0). Nope.
But then its open source so anyone can contribute "features", not that I personally would be using that one you describe

That metadata extension works for me in this sample. How you apply it is up to you

Hi,

Are there any options to deactivate foreign-key constraints, then insert/update/delete table per table using batches, and then reactivate foreign-key at the commit ?
I used a product with that feature which was working at least on mysql and oracle.

thanks
--
Stephane

An L1 cache is an object cache, as defined by the JDO spec. You have an "id" you get the equivalent persistable object. It is used anywhere a persistable object is accessible, and that includes queries if the queries returns a persistable object. The log tells you where objects come from.

Query caching (in DataNucleus, not defined by the JDO spec) is a 3 level process ... generic compilation, datastore compilation, results. Compilations are always cached. Results are only cached when you tell it to cache them, via the query extension datanucleus.query.results.cached
It would make no sense to cache the results of every query for all types of software that use DataNucleus, hence it is off by default.

I've tried to add the extension definition to my package jdo file but either it isn't working as I expect, or I've defined it incorrectly.  This is what I added:

<field name="addPerson" column="ADDPERSON" default-fetch-group="true">
    <extension vendor-name="datanucleus" key="fetch-fk-only" value="true"/>
</field>

With this definition, I was expecting that the object retrieval would basically retrieve the FK value for addPerson from my Manufacturer object.  However, it seems to be trying to retrieve the full object, and puts itself in an endless loop for some reason, causing a StackOverflow.  The datamodel isn't exactly what I listed above, but rather has a bit of a circular design (ie: the AuditPerson actually contains an instance of Audit).   When trying to debug the issue further, I see that DN is calling `dnReplaceFields()` method on my Manufacturer object, but it is difficult to identify which field it is.   I can see the field index value which is causing the endless loop, but can't determine which fieldname the index maps to.  I cannot find the metadata/mapping information where the fieldname is mapped to the field number.  In my case, when I put the debugger on the dnReplaceField in the call stack, I see that it is trying to load field 56, and that this field is causing the infinite loop.  How/where can I identify which fieldname field 56 is mapped to?

Similarly, the DN logs show me what I think is a list of fields that are loaded/not loaded when the object is retreived from cache (loadedFlags) (ex: (id="zzzz" taken from Level 1 cache (loadedFlags="[YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYNNNNNNNNNNN]").  But how do I identify which index is which field?





I would have expected that setting datanucleus.maxFetchDepth=1 would prevent any recursion from happening, but it is still endlessly trying to reload that same field #56.

Removing the `addPerson/modPerson` from the default-fetch-group stops the infinite loop.

I've been encountering something I don't understand with the Query caches and the L1 cache.   I've tried reading the DN docs a few times, but I am still confused by the results I am seeing.

My persistence unit uses the default cache configuration, with no specific settings for the caches identified.  To my understanding then, the queryCache and the queryResult cache should be using soft references with unlimited size.  However, when I add profiling to my DB connection, I see the exact same query being executed multiple times.  For example, I will see the following query executed 4 times in a row:

exec sp_executesql N'SELECT ''izo.model.ActivePerson'' AS DN_TYPE FROM ACTIVEPERSON A0 WHERE A0.TYP = ''izo.model.ActivePerson'' AND A0.ID = @P0 UNION SELECT ''izo.model.Athlete'' AS DN_TYPE FROM ATHLETE A0 WHERE A0.TYP = ''izo.model.Athlete'' AND A0.ID = @P1 UNION SELECT ''izo.model.AthleteAgent'' AS DN_TYPE FROM ATHLETEAGENT A0 WHERE A0.TYP = ''izo.model.AthleteAgent'' AND A0.ID = @P2',N'@P0 bigint,@P1 bigint,@P2 bigint',4319,4319,4319

With a single object relation with the FK at the selected object side there are only 3 things that would be feasible.

1. Load the related object, instantiating it. Put it in the fetch plan for this
2. Don't load the related object and leave for lazy loading. JDO default.
3. Include the PK (FK) field(s) in the SQL, and just store them some where and maybe use them later if required.

DataNucleus (and the JDO API also for that matter) allows options 1 and 2. Specification is explicit.

If you always want to load the related object with just the PK field(s) then you can always mark it as "fetch-fk-only" on that field, in the metadata.
If you only sometimes want to load the related object with just the PK fields, then put the field in the fetch plan for the SELECT, but also remove all other fields of the related object from the fetch plan (so it doesn't add a join to the related object).

Thanks Andy.  Fair enough (embedded vs related).

I forgot to mention the fetch-groups.  Indeed, I have no fetch groups explicity defined - either in the mapping file nor in the API usage.  Everything is using the default fetch group.
But based on your explanation and rereading the DN docs for the N-teenth time, https://www.datanucleus.org/products/accessplatform_5_2/jdo/persistence.html#fetch_groups it clearly identifies that the default fetch group is essentially primitive+ values (ie: int, String, BigDecimal, Date, etc).  So I didn't have any expectation that the Related Objects will be loaded as part of the default fetch group (particularly with maxDepth =1), but was hoping/expecting that at least the FK references would be loaded at the same time to avoid the extra +1 calls just to retrieve the FK values.  

Reading the Kodo docs for the same background, I now see the difference.  In Kodo, they clearly indicate that the object will be left out of the fetch-group, but the FK will be loaded for it.  Whereas in DN, it seems that if the object isn't explicitly added to the fetch-group, the FK will be skipped as well.

I have now added the "addPerson" and "modPerson" to the default-fetch-group, and defined my maxDepth as 1.  And seems to work more inline with expectations.

Thanks,

 

Eric



I'm okay with the extra queries to retrieve the "addPerson" and "modPerson", but don't want the extra DB hit to retrieve their FK values independently.

Is there a way I can configure the defaultFetchGroup to "preload" the FKs during the initial object fetch, without identifying that I want the full objects loaded each time?

Thanks,

Eric

Audit fields "addPerson" and "modPerson" aren't embedded (exist in the same table), they are related (have their own table), using JDO terminology ... or at least without seeing your mapping (XML/annotations) that's what it looks like.

Your mapping definition controls where things are stored, and what is in the default fetch group. Your runtime API usage defines what is in the active fetch group.
What is in the fetch group defines what is fetched ... clue in the name. Kodo maybe didn't have JDO (2) fetch groups.

You rectify it by specifying the appropriate fetching (group) for the appropriate moment in the API usage.

I'm having some strange RDBMS N+1 calls most likely due to my mappings, but am not sure if this is a DN Persistence configuration issue or a mapping issue.  I'm using MSSQL.

My problem is that I have an Object "Person" which has some audit fields "addPerson" and "modPerson".  Both "addPerson" and "modPerson" are simple N-1 relations.  Conceptually like the following:

       //  InheritanceStrategy=New-table


When I retrieve the Person (using a datastore identity), I see an SQL Query which selects the primitive fields but not the embedded Objects.  Something like:
    "Select name, age FROM Person where ID = 123".

When I try to convert to a DTO and read the `addPerson`, and `modPerson` fields, DN then launches the next queries:

• select addPerson FROM Person where ID =123
• select username FROM AuditPerson where ID = <>

• select modPerson FROM Person where ID =123
• select username FROM AuditPerson where ID = <>

So, essentially, because of the "lazy loading", it needs to requery the DB for the 2 FK values in the Person table before it can actually query the AuditPerson table.


Clearly, I suspect that this is due to issues in my mapping definitions.  I am working with old mapping files from Kodo, so I suspect there is something I haven't done correctly.

Questions:
1) Is this normal/expected behaviour?  Is this due to the inheriteance model/hierarchy?  
2) Is this controllable via mapping configurations?
3) Is there a way to reconfigure the model such that I do not need the additonal independent queries to retrieve the FK values for the modPerson and addPerson.


I realize that this is difficult to diagnose without exact details and sample test case, but thought I would ask in the meantime while I try to reproduce the exact behaviour in a test app in case someone has seen/experienced this before.

Thanks,

Eric

Thanks for the tip.  I was looking for a `clearCache` or a `clearMetadata` method or something to that extent that I didn't notice the `unmanage/unload` methods.

I also found the plugin from Dan Haywood for Eclipse that he wrote a long time ago.  Am including the links here for future reference.
- https://www.danhaywood.com/2014/01/23/isis-integration-with-jrebel/
- https://github.com/danhaywood/isis-jrebel-plugin

I'm working with IntelliJ which has a different plugin architecture, and will have to look at this more carefully,  It will lead to many additional questions how to integrate DN with IJ more seamlessly, as currently the enhancement process must be run as a manual task each time a class is modified in IJ.  An IJ plugin that automatically triggers the enhancer for modified classes would be ideal.  

Thanks,

Eric

There is a MetaDataManager for each PMF. It loads metadata when it needs to. There is a method unloadMetaDataForClass to unload a class' metadata if needing to do so.
There is also a method on JDOPersistenceManagerFactory unmanageClass(String className) which will call the underlying MetaDataManager method as well as the associated StoreManager method. Added in DN 4.0.

You could also ask on Apache ISIS support (which uses DataNucleus) because Dan Haywood played around with a JRebel addon to do something related some time ago. No idea if he published it.

I'm working on a fairly large, slow startup legacy application in which I am slowly trying to integrate DataNucleus.  I am also leveraging JRebel, which allows me to modify most classes on the fly without having to restart the application everytime.  For most business oriented logic, this works great.

However, with the DN persistence classes, DN seem to load the metadata into a cache/singleton of some sorts, which prevents me from modifying/reloading a class on the fly, and forces me to stop the application server everytime and relaunch it.  I've sifted through the code and couldn't see anything to enable this, nor did I notice anything in the DataNucleus Metadata Properties so I'm trying to find if there is another mechanism I can use to clear the Metadata cache and have DN reload it on the fly.

Similarly, I would be happy with any process that allows me to modify the persistence class and re-enhance it without needing to restart the application everytime.

Thanks,

Eric

Hi,

I'm running into a similar problem as you where I need to create an ugly JDOHelper.getObjectIdAsLong() hack as well due to some legacy Kodo code.

Did you get something fully functional finally?  Can you share your plugin?  I've looked through the `org.datanucleus.store.rdbms.sql.method.JDOHelperGetObjectIdMethod` implementation, but frankly a little lost by the way it processes everything.  I'm a little unclear on the use of a NumericExpression vs an ObjectLiteral, etc.

Thanks,

Eric

 

Hi,

When 2 queries with the same tenant but different tenantReaders are executed, the second one returns the result of the first one.

The SQL generated is the same for the second query while it should be different.

The query cache key only uses the tenant which may explain the case.

String multiTenancyId = ec.getNucleusContext().getMultiTenancyId(ec);
if (multiTenancyId != null)
{
    queryCacheKey += (" " + multiTenancyId);
}

(no test case yet)

--
Stephane

You are using a JEE connection pool but are setting the resource type as "local" hence DN will try to commit the connection (since it commits local things). Your connection (JBoss JCA) then throws an exception because it wants to commit things, maybe?. Use a local connection if using local ? But then I don't use JEE containers ...

You also have "datanucleus.connectionPoolingType" yet have already specified "connectionFactoryName" to be from JBoss. Either DataNucleus creates the connections to the datastore or JBoss does, but you can't have both, clearly.