Intentional.
JDO Query programmatic "API" with separated items for "from", "filter", "result" etc never had subqueries in JDO1, and when subqueries were added in JDO2 they were specifiable using that programmatic "API" using addSubquery() and variables.
Single string JDOQL also came in at JDO2, with subqueries directly in the single string.

Clearly you could contribute handling, which would involve going to Query class where setFrom is called and parse it for a user including subqueries directly, and store those into the "subqueries" field. Then the primary query and each subquery are each compiled using their own individual JDOQLCompiler like now. No interest here. The other option in setFrom would be to search for the start of a subquery (e.g "SELECT ") and throw an exception informing the user of the error of their ways

Thanks for the link.  I'll take a closer look at them.

I did notice that none of the examples I had found added a subquery filter to a parent query.  I wasn't sure if that was for clarity or by design.

It would seem that appending subquery text into a filter clause of a parent query is not supported.  Would that be accurate?  Is that be a limitation or by design?

Like I said, I tried to step through the parser to identify where it was failing to provide a patch for it, but I was having to trouble  grasping the sequence  of the Lexer parser well enough.

Thanks

Ample examples of JDO subqueries in these tests. Note that none of the DN docs or these examples codes a subquery text into a "filter" clause of the parent query; they either single-string the whole query, or they use a variable for the subquery. They also always package-qualify all class names.

I'm trying to get a complex query working with a subquery in the filter which is failing in the JDOQL Parser.  I have diluted it to a very simple use case with a subselect which is failing as well.  Please see the test-case on github.
I am fairly sure there is something which does not match expected syntax, but I am trying to follow the examples in the DN Docs, but the parser is still failing. 

My query is:

            Query q = pm.newQuery(Person.class);
            q.setFilter("id == (select count(id) from Person p) ");   // fails
            List<Person> results = q.executeList();

and the exception I get is the following:

Does anyone have any ideas what I have done wrong with my subselect?  I've tried tracing through the `JDOQLParser` but without much success.

Thanks,

Eric

I need a distributed L2 caching solution and I've narrowed my options down to the big 3:
- Redis
- HazelCast
- EhCache/Teracotta

Although I've used Teracotta several years ago, I've somewhat eliminated Ehcache/Teracotta due to the lack of knowledge of the platform by the development & operations team, and some of the additional complexities associated with it.  The biggest benefits it provides are:
- per class caching regions
- pinning/unpinning of objects

both of which I'm not sure are critical.  I would definitely like the per-class caching regions, however, I fear the cost to support the Teracotta cluster might be too big.

From what I read, neither Redis nor Hazelcast support either of those in DN.


That being said, I was wondering if anyone had any experience using either Redis or Hazelcast as an L2 cache with DN (JDO) and had recommendations or advice they could share.

Thanks,

Eric

 

The log entries are incomplete.
You still haven't defined what you are invoking. Only by you DEFINING simple classes, and what operation, and seeing persistence code can anyone have a chance of commenting.
The JDO spec tells you what "ignoreCache" can be relied on to do, and it doesn't say that the L1 cache will never be touched. It can be or not, dependent on the implementations choice.

Get the code (datanucleus-rdbms, datanucleus-core) and work through it. Likely PersistentClassROF, ResultSetGetter, ExecutionContextImpl are the classes to look at

Hi Andy,

Below are the logs

Using query to fetch user setting cache false

 

Query "SELECT FROM  User WHERE userId == USERID PARAMETERS Integer USERID FetchPlan [default]" of language "JDOQL" has been run before so reusing existing generic compilation

 

processor Query "SELECT FROM User WHERE userId == USERID PARAMETERS Integer USERID FetchPlan [default]" of language "JDOQL" for datastore "rdbms-mysql" has been run before so reusing existing datastore compilation

 

Executing "SELECT FROM User WHERE userId == USERID PARAMETERS Integer USERID" ...

ManagedConnection OPENED : "org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@3563658 [conn=com.zaxxer.hikari.pool.HikariProxyConnection@12c8578f, commitOnRelease=true, closeOnRelease=true, closeOnTxnEnd=true]" on resource "nontx" with isolation level "read-committed" and auto-commit=false

 

processor Using PreparedStatement "HikariProxyPreparedStatement@1513474198 wrapping com.mysql.cj.jdbc.ClientPreparedStatement: SELECT 

SQL Execution Time = 3 ms

 

ManagedConnection COMMITTING : "org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@3563658 [conn=com.zaxxer.hikari.pool.HikariProxyConnection@12c8578f, commitOnRelease=true, closeOnRelease=true, closeOnTxnEnd=true]"User WHERE userId == USERID PARAMETERS Integer USERID" since the connection used is closing/committing

 

 

Logs while fetching Location by user.location

 

processor Object "User@3a88a807" having fields “Location” fetched from table "`user`"

ManagedConnection OPENED : "org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@41428055 [conn=com.zaxxer.hikari.pool.HikariProxyConnection@4c142df1, commitOnRelease=true, closeOnRelease=true, closeOnTxnEnd=true]" on resource "nontx" with isolation level "read-committed" and auto-commit=false

Using PreparedStatement "HikariProxyPreparedStatement

SELECT { join statement}

SQL Execution Time = 1 ms

Object "User@3a88a807" (id=User$PK@10b8a44e[userId=5303]") taken from Level 1 cache (loadedFlags="[YYYYYYYYYYYYYYYYYYYYYYYNYNYYY]")

 Object “Location@45adce24" (id=“Location$PK@28bb18c5[User=User$PK@4e0db6b8[userId=5303]]") taken from Level 1 cache 

Closing PreparedStatement "org.datanucleus.store.rdbms.ParamLoggingPreparedStatement@55985bfa"

ManagedConnection COMMITTING : "org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@41428055 [conn=com.zaxxer.hikari.pool.HikariProxyConnection@4c142df1, commitOnRelease=true, closeOnRelease=true, closeOnTxnEnd=true]"

ManagedConnection CLOSED : "org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@41428055 [conn=com.zaxxer.hikari.pool.HikariProxyConnection@4c142df1, commitOnRelease=true, closeOnRelease=true, closeOnTxnEnd=true]"

The LOG tells you where objects come from. So post the relevant bits of the LOG for the retrieve, then there is basis for comment. And while you're at it, post the actual PM calls, since ignoreCache only applies in some situations

Hi,

I have two tables Like User and Location and they are mapped by one to many (Location is lazy loaded)

Here are the operations performed (Same persistence manager for both requests)

Ist request-> fetching user from database(persistence manger has set with property "datanucleus.IgnoreCache", true ). So getting user from database by passing cache. And location data is extracted by calling user.location

Changing values in database manually on user and location 

2nd request-> fetching user from database as the cache set to false user changes are appeared but when getting location from user [user.location] old data is appearing. 

As persistence manager cache is set to false, why in lazy loading it is fetching from cache instead of database ?

Yes, well aware, and I will dig deeper into it, since we would like to re-enable L2 caching. 

Another issue we observed when using the L2 cache in our Apache Isis app was that sometimes (?) when an object could not be found in L1 and L2 and was added to L1 the postLoad event was not triggered. Since in Apache Isis dependencies are injected into entities through this event we had cases in which the dependencies were null. In such cases the state of the object was "hollow" (as opposed to "persistent clean"). Also haven't had time to dig deeper into this. 

The l2 cache performs a very simple task (providing access to field values across PMs) and is not the cause of "strange effects". If a field is required and is available in the L2 cache then it populates it from there. If you have some object in some other PM that had the field then it is made available to other PMs. Just enabling/disabling features isn't the way to debug your problems. Disabling the L2 cache will have an impact on speed. Your choice.

Thanks, Andy. It turned out this (and other strange "effects") was seemingly caused by the L2 cache. When we disabled it it went away.

An exception is thrown because it cant find the object in the DB to get the fields from. And it has SQL in the log so you know what it is invoking at that point. So work backwards why it cant find it.

Thanks Andy. Does this mean we should try to load these fields eagerly?

It also seems this only happens under higher load. 

What would be a good place to start looking for why this exception is being thrown? Thankful for any clue, really.

This is an indicator that a field is NOT loaded and something wants it. Only you know your class, and what field you have there, and what is invoked to try to get that field

On Wed, Jul 7, 2021 at 08:24 AM, <mwhesse@...> wrote:

loadUnloadedFieldsInFetchPlan

Wondering about that bit, is this an indicator that the field was loaded previously and has been unloaded? Under which circumstances would that happen? And why does reloading the field fail?

I'm getting the below error (rather "randomly') when loading certain fields of my entities. I don't know where to start looking. Any kind of advice is appreciated.

On DN 5.2 with SQL Server. Let me know if you need more info, thanks.

org.datanucleus.exceptions.NucleusObjectNotFoundException
No such database row
org.datanucleus.store.rdbms.request.FetchRequest#execute(FetchRequest.java:439)
org.datanucleus.store.rdbms.RDBMSPersistenceHandler#fetchObject(RDBMSPersistenceHandler.java:316)
org.datanucleus.state.StateManagerImpl#loadFieldsFromDatastore(StateManagerImpl.java:1542)
org.datanucleus.state.StateManagerImpl#loadUnloadedFieldsInFetchPlan(StateManagerImpl.java:3897)
org.datanucleus.state.StateManagerImpl#isLoaded(StateManagerImpl.java:4112)
domainapp.modules.dcp.dom.quest.dissag.slots.CustomDisaggregationVariableSlot#dnGetdisabled(CustomDisaggregationVariableSlot.java:-1)
domainapp.modules.dcp.dom.quest.dissag.slots.CustomDisaggregationVariableSlot#isDisabled(CustomDisaggregationVariableSlot.java:49)

It is all utterly inconsistent, of course, with javax.persistence developed outside of Oracle (under Eclipse) and so being renamed (just to add pain to its users), and javax.jdo developed outside of Oracle (under Apache) and so not being renamed. Anyway, DataNucleus supports any reasonable persistence spec wherever it is developed.

So on further reading I realized Oracle has just separated out JEE stuff to Eclipse. I had previously misunderstood Oracle's stance to imply that all non "core Java" libraries using javax.* packaging were being separated out to jakarta.* land.

Ah cool. It was my understanding that any 'javax.*' packages would have to be renamed.