Provide BASIS for the error message, such as code, and log entries. That way there is a basis to attempt to answer something that currently has no context.

You could, of course, just get the code and see why that code is hit, but it depends on the context that it was being used ...

Thanks for your reply Andy.

Right, its hard to answer with this less supporting info, I thought if there is something obvious around it that can be changed. I am checking the stacktrace and would get back with more info.

Thanks.

Hello,

After debugging the code I found it is going in the below flow, I ran the queries that were formed, copied them through logs and ran them at DB2, all queries executed correctly but returned no result because data was not present matching that condition. Now, when the flow returns back to our application, We cast result in "Iterable" object, when we call hasNext() it again goes into datanucleus layer and in AbstractQueryResult, query object is found to be null(Hence it throws exception, as per second stacktrace), which is strange as it was set in below stack. Maybe it got garbage collected or some other issue is there as queries were formed properly. Could you please assist or let me know if any further info needs to be furnished.

JDOQLQuery.performExecute(Map) line: 803

JDOQLQuery(Query<T>).executeQuery(Map) line: 1973

JDOQLQuery(Query<T>).executeWithMap(Map) line: 1880

ScrollableQueryResult<E>(AbstractQueryResult<E>).getSizeUsingMethod() line: 350

ScrollableQueryResult<E>.getSizeUsingMethod() line: 567

ScrollableQueryResult<E>(AbstractQueryResult<E>).size() line: 258

ScrollableQueryResult$QueryResultIterator.hasNext() line: 380

ScrollableQueryResult<E>(AbstractCollection<E>).toString() line: 466

String.valueOf(Object) line: 3450

StringBuilder.append(Object) line: 537

JDOQLQuery.performExecute(Map) line: 743

JDOQLQuery(Query<T>).executeQuery(Map) line: 1973

Query<T>.executeWithArray(Object[]) line: 1862

Query<T>.execute() line: 1844

JDOQuery<T>.executeInternal() line: 439

JDOQuery<T>.execute() line: 263

Only you can see it, hence only you can debug it. Check you're not multithreading things. Reproduce it in a testcase. That's all there is.

Hello,

After further debugging we found -  our application goes into Datanucleus layer twice in this flow. First time it creates query, execute it and return result. While returning the result, finally block of JQOQLQuery.java of performExecute() executes mconn.release(), which closes the connection and set query object to null in AbstractQueryResult.java(as ec.getTransaction().isActive() is false, query gets disconnected).
Please find below the code snippet,

JDODQQuery.java

 finally

        {

            mconn.release();

        }

 

public void managedConnectionPreClose()

                                {

                                    if (!ec.getTransaction().isActive())

                                    {

                                        // Non-Tx : disconnect query from ManagedConnection (read in unread rows etc)

                                        qr1.disconnect();

                                    }

                                }

So, in second flow when we go into Datanucleus layer by calling hasNext on returned object(it takes us to hasNext implementation of ScrollableQueryResult), there size() method takes us to AbstractQueryResult, where we get query object as null and exception is thrown.

ScrollableQueryResult.java

        public boolean hasNext()

        {

            synchronized (ScrollableQueryResult.this)

            {

                if (!isOpen())

                {

                    // Spec 14.6.7 Calling hasNext() on closed Query will return false

                    return false;

                }

 

                int theSize = size();

                if (applyRangeChecks)

                {

                    if (theSize < query.getRangeToExcl()-query.getRangeFromIncl())

                    {

                        // Size reached before upper limit of range

                        return iterRowNum <= (query.getRangeFromIncl() + theSize - 1);

                    }

 

                    if (iterRowNum == query.getRangeToExcl()-1)

                    {

                        endIndex = iterRowNum;

                    }

                    // When we are at "query.getRangeToExcl()-1" we have 1 more element

                    return (iterRowNum <= (query.getRangeToExcl()-1));

                }

 

                // When we are at at "size()-1" we have one more element

                return (iterRowNum <= (theSize - 1));

            }

        }

,
I verified ec.getTransaction().isActive() was set to false in previous version(2.0.4) as well. Now, could you please let me know which property I need to set to have the behaviour where query is not set to null as we need to access it later.

I understand I am comparing with very old version, but I have to take reference to make it behave in similar manner. Thank you.

I have zero problem accessing a query result after transaction commit, with scrollable results. You still haven't posted anything resembling a stack trace with current versions, nor a way of reproducing it.

Copy the query results into a standard Java List before closing the results if that is a problem to you

Hello,

Query results are accessible, once the result is returned, we cast it into Iterable. When we call hasNext() on Iterable method, flow goes into Datanucleus layer as hasNext() method is overriden in ScrollableQueryResult.java. 

From there it goes into size method of AbstractQueryResult.java and finally into getSizeUsingMethod method of AbstractQueryResult.java, where the query object is null hence we get below exception. During debugging I found query object of AbstractQueryResult had a value set into it but during connection release/close at the time of returning the result it was set to null and as I explained above, we casted this result into iterable and called hasNext on it, at that time it require query object.

datanucleus.query.resultSizeMethod of "COUNT" is only valid for use with JDOQL or JPQL currently

org.datanucleus.exceptions.NucleusUserException: datanucleus.query.resultSizeMethod of "COUNT" is only valid for use with JDOQL or JPQL currently

at org.datanucleus.store.query.AbstractQueryResult.getSizeUsingMethod(AbstractQueryResult.java:404)

at org.datanucleus.store.rdbms.query.ScrollableQueryResult.getSizeUsingMethod(ScrollableQueryResult.java:567)

at org.datanucleus.store.query.AbstractQueryResult.size(AbstractQueryResult.java:256)

at org.datanucleus.store.rdbms.query.ScrollableQueryResult$QueryResultIterator.hasNext(ScrollableQueryResult.java:380)

at com.ibm.cdb.topomgr.util.extension.iterable.IteratorX.empty(IteratorX.java:79)

Please find below stack trace where we get result and query object is set and later result is returned on which we call hasNext() in our code,

[java.lang.Thread.getStackTrace(Thread.java:1164)]

org.datanucleus.store.query.AbstractQueryResult.<init>(AbstractQueryResult.java:80)]

org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.<init>(AbstractRDBMSQueryResult.java:80)]

org.datanucleus.store.rdbms.query.ScrollableQueryResult.<init>(ScrollableQueryResult.java:82)]

org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getQueryResultForQuery(RDBMSQueryUtils.java:107)]

org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:683)]

org.datanucleus.store.query.Query.executeQuery(Query.java:1973)]

org.datanucleus.store.query.Query.executeWithArray(Query.java:1862)]

org.datanucleus.store.query.Query.execute(Query.java:1844)]

org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:439)]

org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:263)]

OpenQuery.execute(OpenQuery.java:46)]

Also, in order to reproduce it, should I provide dump of query object? or if you could tell what all info is required to reproduce this issue.

https://www.datanucleus.org/documentation/problem_reporting.html

Problem Description and Reproduction

We were able to reproduce the problem in a small standalone program. We took the samples available from
https://github.com/datanucleus/samples-jdo

We picked the "tutorial" sample application and made changes in Main.java and then ran it with DB2 as backend store (version 5.2.3 of datanucleus)
Please find attached modified Main.java

Following is the summary of changes we made and description of what we are attempting:

1. While querying the data, we removed the transaction opening and closing statements
Following lines are commented





2. Following properties are set


loadResultsAtCommit is set to false as we are querying and processing a large amount of data. So, we do not wish to load all the data in memory at one go.
resultSizeMethod is set to "count" for efficiency reason (As stated in data nucleus user guide).

When we are running this sample application with above mentioned changes, we are encountering the following error (while executing hasNext method)

---

Other changes we tried

1. Keeping it as transactional query
          Result: It works
2. Keeping "loadResultsAtCommit" as default, while retaining other 3 properties.
          Result: hasNext() returns true while next() returns null
3. Keeping "loadResultsAtCommit" and "resultSizeMethod" as default, while retaining other 2 properties
          Result: It works

Our analysis

Since, we are migrating from an old version of data nucleus (2.0.4) to version 5.2.3, we looked into data nucleus code to see what changed.
In case of data nucleus 2.0.4, when query is executed (performExecute in JDOQLQuery.java), then connection is released. Since, reference count to this connection becomes zero, it is closed. However, just before a connection is closed, registered callback (look for managedConnectionPreClose in performExecute in JDOQLQuery.java) is invoked.

This callback is empty in old version, while in new version it contains "qr1.disconnect()", which is causing the query object inside query result to become null).
When hasNext is called and resultSizeMethod is set to "count", data nucleus attempt to get the size by constructing a new query but finds existing query object needed for same to be null. Hence, the error.

Summary
Since, we are migrating a legacy application, this error got us confused as this combination of query properties use to work correctly with old version of data nucleus.

Also, We are executing a query without transaction and processing a large number of records which may take a while. Processing of records do also involve opening of several transactions. So, we are not sure of putting this query inside a transaction (Request for a general guidance in this regard).
Also, it seems if we decide to remove loadResultsAtCommit and resultSizeMethod, that may impact performance.

A overall guidance in terms of how either how we can adapt according to this changed behavior of data nucleus or if we could some apply some patch on data nucleus is highly appreciated.

That is not a "testcase". A "testcase" is something that, when run, returns SUCCESS, FAILURE or ERROR dependent on how it performs and how it is coded based on expectations.

That aside, you decide to run non-transactional and then use an extension based on being transactional (loadResultsAtCommit), which makes no sense.

You also run non-transactional and expect any connection to be held open until the end of PM, without specifying datanucleus.connection.nontx.releaseAfterUse.
https://www.datanucleus.org/products/accessplatform_5_2/jdo/persistence.html#_nontransactional_context



Clearly better error messages could be provided, but then people could contribute such handling.

Hello Andy,

In the same issue, we minimized the extensions and ran the query with below two extensions but our sample program is going into infinite loop in execute method. Please find attached the stacktrace. Since stacktrace was big only limited lines were printed on console, but shows the infinite loop sequence in method call inside it.

PersistenceManager pm = pmf.getPersistenceManager();

    Query q=pm.newQuery("SELECT FROM " + Product.class.getName() + " WHERE price < 150.00 ORDER BY price ASC");

 

q.addExtension( "datanucleus.rdbms.query.resultSetType","scroll-insensitive"); 

q.addExtension( "datanucleus.query.resultSizeMethod", "count");

    List<Product> products = (List<Product>)q.execute();

Since you are the only person who sees this, I suggest you get the code and work it out