Re: Migration from Kodo: Any way to make it store foreign keys to later get already loaded objects from cache (for performance)?


dosiwelldi
 

Hi again, 2 more things:

1) Parser Bug

I think I found a bug in the JDOQL query parser: booleans fields in parentheses lead to javax.jdo.JDOUserException: Method/Identifier expected

While these are ok (`active` is a boolean field on Class enterprise):
pm.newQuery(Enterprise.class, "active");
pm.newQuery(Enterprise.class, "!active");
pm.newQuery(Enterprise.class, "(!active)");

This one will throw an exception:
pm.newQuery(Enterprise.class, "(active)");     //Method/Identifier expected

Stack trace:
    [junit] Method/Identifier expected at character 9 in "(active)"
    [junit] org.datanucleus.store.query.QueryCompilerSyntaxException: Method/Identifier expected at character 9 in "(active)"
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processPrimary(JDOQLParser.java:802)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processUnaryExpression(JDOQLParser.java:643)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processMultiplicativeExpression(JDOQLParser.java:569)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processAdditiveExpression(JDOQLParser.java:540)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processRelationalExpression(JDOQLParser.java:454)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processAndExpression(JDOQLParser.java:437)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processExclusiveOrExpression(JDOQLParser.java:423)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processInclusiveOrExpression(JDOQLParser.java:409)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processConditionalAndExpression(JDOQLParser.java:395)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processConditionalOrExpression(JDOQLParser.java:376)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.processExpression(JDOQLParser.java:365)
    [junit]     at org.datanucleus.query.compiler.JDOQLParser.parse(JDOQLParser.java:88)
    [junit]     at org.datanucleus.query.compiler.JavaQueryCompiler.compileFilter(JavaQueryCompiler.java:600)
    [junit]     at org.datanucleus.query.compiler.JDOQLCompiler.compile(JDOQLCompiler.java:103)
    [junit]     at org.datanucleus.store.query.AbstractJDOQLQuery.compileGeneric(AbstractJDOQLQuery.java:392)
    [junit]     at org.datanucleus.store.query.AbstractJDOQLQuery.compileInternal(AbstractJDOQLQuery.java:450)
    [junit]     at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:263)
    [junit]     at org.datanucleus.store.query.Query.executeQuery(Query.java:1936)
    [junit]     at org.datanucleus.store.query.Query.executeWithArray(Query.java:1864)
    [junit]     at org.datanucleus.store.query.Query.execute(Query.java:1846)
    [junit]     at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:439)
    [junit]     at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:263)


2) query.getFetchPlan().setFetchSize(1) will still retrieve the whole table

In short:
I probably understood wrongly what query.getFetchPlan().setFetchSize(1) does. I wanted to prevent queries from loading whole tables initially, to just get rows as needed. But I see in the debugger that JDOFetchPlan.getFetchSize() is never called for my code so it is probably not meant for that.

In long:
I wrote a script that tries to extract most of our queries into a unit test to see if they still work. I only care if they compile so I do not iterate over the result collection. Kodo takes about 1 minute to execute all of them. If I run the same test with DN I get an out of memory exception (I think from junit that tries to cache the log file). Without logging it runs but will never finish.

I then added query.getFetchPlan().setFetchSize(1) to each query so it should only get one row. But it still gets the whole table.

//Example: Takes 68s with DN and reads 710k rows:
Query query = pm.newQuery(OperationalEvents.class);
query.getFetchPlan().setFetchSize(1);
query.execute();

I now added query.setRange(0, 1) to all tests as a workaround. This works fine for my test. But we have some queries in the real code that ask for a lot of data but then only access the first row... Will be fun to try to find them all.

Log after many seconds: now getting rows 101542 and 101543:
    [junit] 2021-01-28 19:36:20,481 DEBUG [main] DataNucleus.Persistence - Retrieved object with OID "xxx.yyy.OperationalEvents:101542"
    [junit] 2021-01-28 19:36:20,481 DEBUG [main] DataNucleus.Cache - Object with id "xxx.yyy.OperationalEvents:101542" not found in Level 1 cache
    [junit] 2021-01-28 19:36:20,481 DEBUG [main] DataNucleus.Cache - Object with id "xxx.yyy.OperationalEvents:101542" not found in Level 2 cache
    [junit] 2021-01-28 19:36:20,481 DEBUG [main] DataNucleus.Cache - Object "xxx.yyy.OperationalEvents@394851d7" (id="xxx.yyy.OperationalEvents:101542") added to Level 1 cache (loadedFlags="[NNNNNNN]")
    [junit] 2021-01-28 19:36:20,481 DEBUG [main] DataNucleus.Cache - Object "xxx.yyy.OperationalEvents@394851d7" (id="xxx.yyy.OperationalEvents:101542") added to Level 2 cache (fields="[0, 1, 2, 3, 4, 5, 6]", version="")
    [junit] 2021-01-28 19:36:20,482 DEBUG [main] DataNucleus.Persistence - Retrieved object with OID "xxx.yyy.OperationalEvents:101543"
    [junit] 2021-01-28 19:36:20,482 DEBUG [main] DataNucleus.Cache - Object with id "xxx.yyy.OperationalEvents:101543" not found in Level 1 cache
    [junit] 2021-01-28 19:36:20,482 DEBUG [main] DataNucleus.Cache - Object with id "xxx.yyy.OperationalEvents:101543" not found in Level 2 cache

Join main@datanucleus.groups.io to automatically receive all group messages.