Date   

Re: PersistentClassROF#getObject method performs a shallow search so not finding all subclasses

Page bloom
 
Edited

I found the cause eventually!

This part of the error is possibly misleading:

Really you need a discriminator to help identifying the type
It seems to suggest that you need to use a discriminator (which we already are) so that could lead the developer to check their metadata etc.,

The real problem was that metadata was missing due to two package.jdo files being provided for the same package but having different classes in them - result of a split up/refactor that ended up reusing the same package name for different classes - some in the main app and some that were migrated to a shared library, hence at runtime two package.jdo files were present "in" the same package (same logical package but two separate package.jdo files).

The root cause was that the lookup of the class name using the discriminator value failed - because the metadata was not present.

Perhaps I can look at making the error a more accurate description of the cause - or maybe it needs two separate error messages:
  1. One for when discriminator column is really not configured in the metadata
  2. When discriminator column is configured in the metadata but the specific discriminator value could not be mapped to a known PC


PersistentClassROF#getObject method performs a shallow search so not finding all subclasses

Page bloom
 
Edited

We've noticed a problem where, during object retrieval of an object with a class that is 2 levels derived from an abstract base class, DN displays the message:

[WARN ] 05:33:20.915 Retrieve - Found type=class com.sas.framework.BlahBlah but abstract and more than 1 concrete subclass ([com.sas.BlahLevel1Cls1, com.sas.BlahLevel1Cls2, com.sas.BlahLevel1Cls3]). Really you need a discriminator to help identifying the type. Choosing class com.sas.BlahLevel1Cls3

This was unexpected because:
  • A discriminator is configured in the abstract base class
  • A discriminator value is provided in each of the derived classes
  • The list of concrete subclasses is only a small subset of the classes in the model
  • It is only showing the list of subclasses immediately extending the abstract base class
Looking at PersistentClassROF#getObject we noticed that the call to getSubclassesForClass passes 'false' for the 'includeDescendants' parameter - which would prevent it from finding subclasses beyond the classes that immediately extend the abstract base class.

            String[] subclasses = this.ec.getMetaDataManager().getSubclassesForClass(pcClassForObject.getName(), false);
 
 Is this intentional? How will it ever find the full range of subclasses if this parameter is false?


Re: Generated dnProvideFields method appears to not handle case where fieldNumbers array is null (i.e. class has no attributes) in DN 5.1.6

Page bloom
 

I have attached a very simple, two class, test case which reproduces the issue of having no attributes in the base class.

A simple one class scenario with no attributes in the base class also reproduces the problem but that's a bit of a pointless model.


Re: What's the best way to write a JDOQL query to compare dates that doesn't give a warning about using a transient as parameter in query?

Page bloom
 
Edited

Ah, Doh! I was looking at the wrong parameter. I had assumed it was the Date parameter causing the trouble but it was another seemingly innocent parameter we had been passing into the query - not shown above in the 'cut down' version of the query.

Not sure how the full query in question in the app ever worked .... hmmmm. I think it was only used for some data migrations that never went ahead so that probably explains how it "not working" was not a problem in the app.

 

 


Re: What's the best way to write a JDOQL query to compare dates that doesn't give a warning about using a transient as parameter in query?

Andy
 

The best way to not get a message about passing in a transient persistable object is not to pass in a transient persistable object.
Doing what you say above about passing a Date (which is not a persistable object) does not result in such a warning ... for me.


What's the best way to write a JDOQL query to compare dates that doesn't give a warning about using a transient as parameter in query?

Page bloom
 
Edited

UPDATE: Turns out this was a problem with another parameter, not the date parameter, which was in fact a transient persistable class. I had left this other parameter out of the example, assuming the date parameter was the issue :) Date comparisons work fine in JDOQL/DataNucleus right out of the box!
 

What's the best way to compare a date attribute of a PC with a specified Date object (which isn't persistent) without getting this warning:
 
[WARN ] 13:17:01.397 Query - Attempt to use transient object as parameter in query. Not supported, so using NULL for parameter value
 
This is a bit of a construed example but here goes...
 
class Loaned
{
    Date date;      // java.util.Date - date the Book was loaned on
}
 
class Book
{
      Loaned loaned;
}
 
So to find all Books loaned before a given date  
 
Collection<Book> findBooksLoanedBefore(Date date)
{
    PersistenceManager pm = getPm();
 
    Query q = pm.newQuery(Book.class);
    q.setFilter("loaned.date <= :date");
 
    return new JDOQueryResultCollection(q, q.execute(date));
}
 
Perhaps persistent Date objects attributes of a PC can't be compared with a transient Date object like this.
 
If not, what's the best way to perform a date comparison that takes advantage of the DB's indexing - I don't want to have to do an in memory comparison because there could be millions of objects.
 
Is the best way to store dates as long's (eg., the result of Date.getTime() ) and then to all comparisons as simple maths comparisons?
 
In this case we'd pass in date.getTime() as the parameter to the query.
 
That seems a bit primitive/low level - I would hope there's a higher level way of achieving this without resorting to storing and comparing ints.


Re: Generated dnProvideFields method appears to not handle case where fieldNumbers array is null (i.e. class has no attributes) in DN 5.1.6

Page bloom
 

This issue is independent of previous interface issues that we recently discussed.

In this scenario the base class implements Serializable but I removed that "implements" relationship as a test and the problem remains - i.e. simple 3 class hierarchy with no references to other classes or interfaces.

I'll look up checkForSchemaUpdatesForFieldsOfObject and see if I can attack the issue from in there.


Re: Generated dnProvideFields method appears to not handle case where fieldNumbers array is null (i.e. class has no attributes) in DN 5.1.6

Andy
 

The StateManager and enhancement contract are not the place to change anything.
You need to prevent calling dnProvideFields with crap. i.e checkForSchemaUpdatesForFieldsOfObject (i.e where you have interfaces and some discovered implementations).
This is only of relevance there.


Re: Generated dnProvideFields method appears to not handle case where fieldNumbers array is null (i.e. class has no attributes) in DN 5.1.6

Page bloom
 
Edited

Passing an empty fieldNumbers native array does prevent the dnProvideFields method from throwing an IllegalArgumentException 

I can raise a pull request for a fix like this if you think it's reasonable:

StateManagementImpl.java lines 2507-2517 
 
            try
            {
+                // In case class has no attributes pass empty array instead of null
+                if (fieldNumbers == null)
+                    fieldNumbers = new int[0];
+
                // This will respond by calling this.providedXXXFields() with the value of the field
                myPC.dnProvideFields(fieldNumbers);
            }
            finally
            {
                currFM = prevFM;
            }

It is probably a simpler fix than changing the generated dnProvideFields method but making the change there might be more "correct" - and it might even reduce the size of the generated method, possibly speeding up enhancement of large persistent models (but maybe not significantly)


Re: Generated dnProvideFields method appears to not handle case where fieldNumbers array is null (i.e. class has no attributes) in DN 5.1.6

Page bloom
 

I found the code that's adding the dnProvideFields methods during BCE:

org.datanucleus.enhancer.methods.ProvideFields.java

The execute method is doing the comparison with null here:

 

    public void execute()

    {

        visitor.visitCode();

 

        Label l0 = new Label();

        visitor.visitLabel(l0);

        visitor.visitVarInsn(Opcodes.ALOAD, 1);

        Label l1 = new Label();

        visitor.visitJumpInsn(Opcodes.IFNONNULL, l1);

        visitor.visitTypeInsn(Opcodes.NEW, "java/lang/IllegalArgumentException");

        visitor.visitInsn(Opcodes.DUP);

        visitor.visitLdcInsn("argment is null");

        visitor.visitMethodInsn(Opcodes.INVOKESPECIAL, "java/lang/IllegalArgumentException",

            "<init>", "(Ljava/lang/String;)V");

        visitor.visitInsn(Opcodes.ATHROW);

It's all in Java VM machine language (obviously much more efficient than performing a compile phase during enhancement) which makes for interesting reading ;)

I'm wondering if it's better to just change StateManagerImpl so that it passes an empty Array into dnProvideFields rather than changing the enhancer...


Generated dnProvideFields method appears to not handle case where fieldNumbers array is null (i.e. class has no attributes) in DN 5.1.6

Page bloom
 

In this class diagram you can see the abstract base class doesn't have (nor requires) any of it's own attributes. The FieldManager class could have been an interface apart from the fact that:
  • we wanted all classes in this hierarchy to be stored in a single database table
  • we plan to add some stats attributes later and these would definitely go into this base class



In DN 5.1.6 we get the following exception:

java.lang.IllegalArgumentException: argment is null
at com.sas.framework.filemgr.FileManager.dnProvideFields(FileManager.java)
at org.datanucleus.state.StateManagerImpl.provideFields(StateManagerImpl.java:2510)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.checkForSchemaUpdatesForFieldsOfObject(RDBMSPersistenceHandler.java:667)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:118)
at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:4535)
at org.datanucleus.state.StateManagerImpl.flush(StateManagerImpl.java:5735)
at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:490)
at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:366)
at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:191)
at org.datanucleus.state.StateManagerImpl.providedObjectField(StateManagerImpl.java:1823)

The DN 5 StateManagementImpl class is passing a null fieldNumbers array to the method: com.sas.framework.filemgr.FileManager.dnProvideFields, a generated method added during byte code enhancement.

           currFM = fm;
            try
            {
                // This will respond by calling this.providedXXXFields() with the value of the field
                myPC.dnProvideFields(fieldNumbers);       <------ fieldNumbers array is null because this base class has no attributes - which should be fine
            }
            finally
            {
                currFM = prevFM;
            }

The call is fine, however the problem is in the generated code in the PC's dnProvideFields method. It may not be handling the case where it passed in a fieldNumbers array which is null, however null seems like a valid case if the class has no attributes.

By adding an "int dummy" attribute to the base class the issue goes away which indicates that the issue is related to having no attributes in a base class.

Where would we find the code that is responsible for generating the dnProvideFields method in the DN code base?


Re: Issue with a relationship to an interface

Page bloom
 
Edited

The problem seemed to be intermittent when we had the implementing class in a different package. Once we moved it into the same package it didn't happen again - but on another dev machine it worked fine while in the different package - strange.

Anyway, the DataNucleus docs do mention some non deterministic behaviour if the metadata of implementations is not experienced prior to experiencing the class with the relationship to the interface. The docs say to avoid this non deterministic behaviour specify the implementation classes in the metadata (which we have now done)

eg.
field-type="com.sas.blah.blah.ImplementationClass"

Everything seems to go just swimmingly now :) (because terms from the 1920s need a revival...)

I still think the change in the pull request would be useful to anyone else who finds themselves in this situation as it helps to narrow down the exact class+attribute causing the issue.


Re: Issue with a relationship to an interface

Page bloom
 

When I ran my app against a version of dn-rdbms with this change (pull request created) it revealed, for the first time, which PC class + attribute was causing the issue that resulted in the exception.

So at least this change enables me to drastically reduce the scope of my search for the root cause :)  (which is still a bit of a mystery at the moment)


Re: Issue with a relationship to an interface

Andy
 

It would be best to work out where the problem is in your mapping+classes and hence why you get some problem. Once you see that then you can put a trap at the root cause. Aka work backwards from where you are looking


Re: Issue with a relationship to an interface

Page bloom
 

I'm now actually getting this exception in the main app during start up.

I'm not sure if I've got the logging setup correctly (recently switched to Log4j 2 and unsure how to use the DataNucleus categories in that version) but the exception is thrown without giving any indication which class or attribute is causing the issue:

Failed to generate new Mapping of type org.datanucleus.store.rdbms.mapping.java.InterfaceMapping, exception : org.datanucleus.store.rdbms.mapping.java.SerialisedMapping cannot be cast to org.datanucleus.store.rdbms.mapping.java.PersistableMapping
java.lang.ClassCastException: org.datanucleus.store.rdbms.mapping.java.SerialisedMapping cannot be cast to org.datanucleus.store.rdbms.mapping.java.PersistableMapping
at org.datanucleus.store.rdbms.table.ColumnCreator.createColumnsForField(ColumnCreator.java:308)
at org.datanucleus.store.rdbms.mapping.java.ReferenceMapping.createPerImplementationColumnsForReferenceField(ReferenceMapping.java:488)
at org.datanucleus.store.rdbms.mapping.java.ReferenceMapping.prepareDatastoreMapping(ReferenceMapping.java:230)
at org.datanucleus.store.rdbms.mapping.java.ReferenceMapping.initialize(ReferenceMapping.java:104)
at org.datanucleus.store.rdbms.mapping.java.InterfaceMapping.initialize(InterfaceMapping.java:52)
at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMapping(MappingManagerImpl.java:667)
at org.datanucleus.store.rdbms.table.ClassTable.manageMembers(ClassTable.java:549)
at org.datanucleus.store.rdbms.table.ClassTable.manageClass(ClassTable.java:455)
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3347)
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2961)
at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:118)
at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1672)
 
I am thinking of ways to change the DN source to give an indication of the class/field that is causing the issue to make it easier for people to debug their particular issues.

The code where the above exception is raised is in:

ColumnCreator#createColumnsForField (line 308)

                    ((PersistableMapping)container).addJavaTypeMapping(refDatastoreMapping);   <--- Cast fails

Once the exception is raised it is handled at a higher level where, presumably, specific field information is no longer available.

Would it be viable to test if container was an instanceof 'PersistableMapping' and throw a DN exception if not, passing in the field data?

eg 

if (container instanceof PersistableMapping) 
    ((PersistableMapping)container).addJavaTypeMapping(refDatastoreMapping);
else
    throw new NucleusUserException("Cannot create column for field " + mmd.getFullFieldName());

or is it best to wrap it in a try/catch?

try
{
    ((PersistableMapping)container).addJavaTypeMapping(refDatastoreMapping);
}
catch(ClassCastException cce)
{
    throw new NucleusUserException("Cannot create column for field " + mmd.getFullFieldName(), cce);
}


Re: Issue with a relationship to an interface

Page bloom
 

Ah doh!

That stack dump must have been when I reverted to 4.x to test it in that version and I've now gone back to 5.x.

RDBMSMappingManager is obviously a Version 4.x class and renamed to something else in 5.x.

Sorry, my bad!


Re: Issue with a relationship to an interface

Andy
 

I meant the class that YOUR stack trace quotes. In current codebase there is no such class (but there is MappingManagerImpl). In whatever version you're using there is that class.


Re: Issue with a relationship to an interface

Page bloom
 

On Wed, Mar 21, 2018 at 12:00 am, Andy wrote:
RDBMSMappingManager
I can't find that specific class. Did you mean MappingManagerImpl in the *.rdbms package?


Re: Issue with a relationship to an interface

Page bloom
 

I'm planning to create a minimalist class model that demonstrates the issue - but, as usual with our massive model, it will take some time to work out the minimalist chunk of it that exhibits the issue.

I was just putting it out there early in case someone has seen this before and dealt with it.

It seems to be when reading/processing the metadata - not when attempting to build the schema because it happens in the unit test's setUp method whether or not

datanucleus.generateSchema.database.mode=create

is present or commented out.

This happens with 5.1.6 and 5.1.7 and so I rolled back to 4.x and it happens with that also.


Re: Issue with a relationship to an interface

Andy
 

So demonstrate it or otherwise get the code for datanucleus-rdbms and debug from RDBMSMappingManager as to why it is choosing that type

381 - 400 of 446