|
[
Permalink
| « Hide
]
lucols added a comment - 19/Dec/05 12:31 AM
I found the problem on linux as3, Jboss2.7, jboss4..0.3 ,mysql5.0.16 ,jdk1.4..2.08; but no found on windows with same versions.
Same problem here on linux and jdk 1.5, but only when I use the client vm. The server vm works fine!
Maybe this gives a hint where the problem comes from. did you upgrade CGLIB with the Hibernate dist one?
Yes, my classpath contains cglib-2.1.3.jar
Neither cglib2.1.2 nor cglib2.1.3 , the problem were happen. the problem is not happen at the first, but the pojo be load again and again (about ten times) , the exception happened , and all operation relate to hibernate invalid after that. I think this problem relate with cglib , because cglib implement is different betweem linux and windows.
who can use other someting repalce the cglib function in hibernate ? Linux/Java 1.4 Hibernate 3.1 CGLIB 2.1.3
Also appears after a few tries. Any of you guys found a workaround? Some source code hack maybe. Its a bit critical as running old proprietary db is really expensive for the project. Mysql 5.0 looks good and is running stable. It just so happened that we are dependent on hibernate. Looks bad coz I have to turn of the old db by december and move to mysql by january. Also appears sometimes on a Windows box(Tomcat) connecting to a linux box(mysql). You should have decoupled your framework from hibernate. But then again you have HQL to worry about. Im pretty dependent to it too.
Did any of you guys even try recompiling Hibernate on the JDK you are using?
Christian, same effect.
So the next step is to create a very very very simple test case (one mapping file, one class, etc.) that triggers the problem and that we can run to identify the problem, unless somebody else feels able to do this. Attach it to the JIRA issue.
Ok, I would try to make one then. Including the mysql source script.
Not unless, someone has a small sample ready, I would have to take a while, im still trying to figure out which part of my app's hibernate statements causes the error.
To James Gatdula:
I posted this before: This error does NOT occur with the server virtual machine, only with the client vm. Maybe some of you can try this out and confirm it. It might help you while the error has not been fixed. I can confirm that it does not happen on the server VM:
# java -server -version java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode) Running Tomcat 5.0.27, Hibernate 3.1, CGLib 2.1.3 Hi Andreas
I can confirm that it happens on the server VM. Only difference from your setup is Im using: java version "1.4.2_09" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_09-b05) Java HotSpot(TM) Server VM (build 1.4.2_09-b05, mixed mode) Running under linux/jdk1.5.0_03, I can repro this in rc3, but not in rc2.
This patch fixes it for me:
diff -ru hibernate-3.1release/src/org/hibernate/proxy/CGLIBLazyInitializer.java hibernate-3.1/src/org/hibernate/proxy/CGLIBLazyInitializer.java --- hibernate-3.1release/src/org/hibernate/proxy/CGLIBLazyInitializer.java Fri Dec 23 13:39:30 2005 +++ hibernate-3.1/src/org/hibernate/proxy/CGLIBLazyInitializer.java Fri Dec 23 13:40:27 2005 @@ -117,8 +117,7 @@ en.setUseCache( false ); en.setInterceptDuringConstruction( false ); - en.setCallbackTypes( CALLBACK_TYPES ); - en.setCallbackFilter( FINALIZE_FILTER ); + en.setCallbackType( MethodInterceptor.class ); en.setSuperclass( interfaces.length == 1 ? persistentClass : null ); en.setInterfaces( interfaces ); Thanks Michael . your patch resolved my problem - the exception 'NoSuchMethodError..getHibernateLazyInitializer' not throw again, and the web applicateion not break down . Unfortunately, now , another problem coming, when i query some presister persist object , report error:
ERROR [org.hibernate.LazyInitializationException] could not initialize proxy - the owning Session was closed org.hibernate.LazyInitializationException: could not initialize proxy - the owning Session was closed at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:56) at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:98) at org.hibernate.proxy.CGLIBLazyInitializer.intercept(CGLIBLazyInitializer.java:163) at com.noww.opts.entity.Contact$$EnhancerByCGLIB$$a0cd1bca.finalize(<generated>) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83) at java.lang.ref.Finalizer.access$100(Finalizer.java:14) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) patch befor , this error not happen, now , why? . ( I get session like this: public class HibernateSessionFactory { private static Log log = LogFactory.getLog(HibernateSessionFactory.class); private static SessionFactory sessionFactory; public synchronized static Session openSession() throws HibernateException { if (sessionFactory == null) { init(); } return sessionFactory.openSession(); } private static void init() { try { sessionFactory = new Configuration().configure(). buildSessionFactory(); } catch (MappingException e) { log.error(e); } catch (HibernateException e) { log.error(e); } } }) I change the hibernate 3.1 release CGLIBLazyInitializer.java as Michael 's patch , and run build.bat to get the new hibernate3.jar , ohters not be changed.
Run env: jboss 3.2.7(default ), cglib2.3.1 jdk1.4.2.0.8 ,mysql 5.0.16, linux AS 3. Can nobody attach a simple test that shows the problem?
I just had this problem at work with Hib3.1 on Linux yesterday. The problem is definitely intermittent.
It *should* happen for anything that calls EntityType.resolveIdentifier(), which should be just about any application. This only seems to happen on Linux. I was not able to repro it on Win-doze. Haven't tried Solaris. @Christian: When I'm back from Christmas holidays (~27.12.) I'll try to prepare a testcase.
It looks like a problem with concurrency. "getHibernateLazyInitializer()" method is called on proxy before interceptor ("CGLIBLazyInitializer") is set.
"de.schildbach.game.integration.HibernateGamePlayer$$EnhancerByCGLIB$$afecb986.getHibernateLazyInitializer(<generated>) " is generated this way: public HibernateLazyInitializer getHibernateLazyInitializer(){ if( interceptor != null ){ return interceptor.intercept( .... ); }else{ return super.getHibernateLazyInitializer();//"java.lang.NoSuchMethodError" if "interceptor" is null; } } patch in interceptor implementation must not help too:
"- en.setCallbackTypes( CALLBACK_TYPES ); - en.setCallbackFilter( FINALIZE_FILTER ); " It will generatete the same code except "finalize" (nothing will be generated for this method) Probably it was not a good idea to generate "super.myMethod", it must be better to throw "IllegalStateException("interceptor is not set for abstract method")" I am not sure, but it looks like a concurrency problem in global cache (if "initializeFromCache" is related to L2 cache ). Try to dissable it.
I have disabled the second level cache, but to no avail: After executing the same query for a few times the exception is thrown.
Proxies are not thread safe and it looks like a problem with concurrent access, make sure application code is thread safe (proxies or hibernate session stored in http session is a typical problem). But I am not sure, post more stack traces if available and application code related to session management .
Thread implementations are different (JVM on some linux kernels uses child process and shared memory to implement multithreading, it can depend on JVM settings and implementation) and can produce many anomalies if code is not thread safe. The right way is to make application code thread safe, workarounds can help to hide a problem, but not to solve it.
Switching to the 'server' JVM (I'm using RHEL3 and JDK1.4.2_09) seems to have fixed this for me. We're switching to RHEL4 anyway because there are other problems with the Sun JVM on RHEL3.
I took a look at the code for multi-threaded access to proxies and I didn't find anything obviously wrong. There are definitely no Sessions stored in HttpSessions, but there may be a proxy that is accessed by multiple threads. Juozas, Can you be a little more precise about where you think the thread safety problem is? Is it in determining whether the proxy is initialized or not? Juozas, Oh... you did list the method. Sorry.
Juozas, any idea of why this issue is starting to pop up know ? Why didn't users see this previously ?
We will fix error message in next release, but it will not solve this problem (probably it will help to debug, current error message confused myself). Temprory workarouind can be a synchronized filter, it will not scale, but it must work.
"getProxy" method implementation was changed. This change is correct, probably 3.05 code forces JVM to replicate shared memory or to flush internal cache (this stuff is random if access to shared data is not synchronized)
Hi Juozas Baliuka,
How do you implement the synchronized filter? The patch sorta made it work on my side... but I also ended up with the owning session closed like errors afterwards. Since my system is using interceptors for http request in and out, i open an close sessions at the end points. So just to make the system survive, I just silenced the error, but I still need to abandon the connections, so I set the connection pool to automatically abandon connections after a certain time. My last issue now is that when a good connection was opened but not necessarily used right away and the connection was forced to be abandoned in between I get another new error. Its difficult and sad since my design was not prepared for such an error hitting the very heart of my system. Hi Max Rydahl Andersen, My personal reason is that I still did some rnd test on mysql 5.0 itself. When my company gave the order to drop the old propietary database. I needed a solution. If you are using "front controler servlet" then you can use "javax.servlet.SingleThreadModel" http://www.javaworld.com/javaworld/jw-07-2004/jw-0712-threadsafe-p3.html as hack or map fiter with synchronized "doFilter" method http://www.onjava.com/pub/a/onjava/2001/05/10/servlet_filters.html. It must help if you assign persistent objects to servlet fields (directly or inderectly).
This workaround must scale better if you store proxies in http session (the right way is to replace all session object attributes with primary keys) public void doFilter (ServletRequest request, ServletResponse response, FilterChain chain){ User user = request.getSession().getAttribute(USER_KEY); if(user ! = null){ synhronized(user){ chain.doFilter(request,response); } }else{ chain.doFilter(request,response); } } If this is "Memory Model" / multithreading problem (I am not sure, it is hard to reproduce this random stuff) then it must help. If this is "Memory Model" problem then session wrapper idea must help too
http://forum.hibernate.org/viewtopic.php?t=946052&start=15. If this doe's not help then I will try to reproduce it myself (I see it must not be easy to reproduce in simple test case and it will take time). Hm, a session filter is a specialty of web applications, right? My application has a client-server architecture with a swing client und RMI
for the communication with the server process. The server is multi threaded. How do I synchronize an application like that? It is possible to implement interception using dynamic proxies for RMI methods to make it "transparent" or just to declare all remote method implementations "synchronized", EJB container can help too (session beans). But plain RMI server application is stateless (session per method and no shared data like private fields, it must not have synchronization issues), is not it ?
I thought so, I just had to ask.
Maybe I could "fake" my own version of hibernate for linux. Which method would I have to synchronize? It does not make sence to hack hibernate if this is synchronization issue, just synchronize all remote method implementations (add "synchronized keyword"). It must solve synchronization problems, but it will not solve concurrency problems related to transactions (memory leaks are possible if you share persistent data using RPC object fields) . It must not be so hard to refactor application too.
I don't think this is a synchronization issue; I am getting it running single threaded on Linux. My stack trace (from rc3; this doesn't happen in rc2) looks like:
java.lang.NoSuchMethodError: com.xxx.xxx.model.Level4.getHibernateLazyInitializer()Lorg/hibernate/proxy/LazyInitializer; at com.xxx.xxx.model.Term$$EnhancerByCGLIB$$7d77adec.getHibernateLazyInitializer(<generated>) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:274) at org.hibernate.type.EntityType.resolve(EntityType.java:303) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842) at org.hibernate.loader.Loader.doQuery(Loader.java:717) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:223) at org.hibernate.loader.Loader.loadCollection(Loader.java:1916) at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:71) at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:520) at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:60) at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1493) at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:344) at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86) at org.hibernate.collection.PersistentSet.iterator(PersistentSet.java:138) at org.hibernate.type.CollectionType.getElementsIterator(CollectionType.java:196) at org.hibernate.type.CollectionType.getElementsIterator(CollectionType.java:188) at org.hibernate.engine.CascadingAction.getAllElementsIterator(CascadingAction.java:266) at org.hibernate.engine.CascadingAction.access$100(CascadingAction.java:21) at org.hibernate.engine.CascadingAction$2.getCascadableChildrenIterator(CascadingAction.java:52) at org.hibernate.engine.Cascade.cascadeCollectionElements(Cascade.java:288) at org.hibernate.engine.Cascade.cascadeCollection(Cascade.java:185) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:160) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:108) at org.hibernate.engine.Cascade.cascade(Cascade.java:248) at org.hibernate.engine.Cascade.cascade(Cascade.java:223) at org.hibernate.event.def.DefaultDeleteEventListener.cascadeBeforeDelete(DefaultDeleteEventListener.java:220) at org.hibernate.event.def.DefaultDeleteEventListener.deleteEntity(DefaultDeleteEventListener.java:169) at org.hibernate.event.def.DefaultDeleteEventListener.onDelete(DefaultDeleteEventListener.java:110) at org.hibernate.impl.SessionImpl.fireDelete(SessionImpl.java:712) at org.hibernate.impl.SessionImpl.delete(SessionImpl.java:706) at org.hibernate.engine.CascadingAction$2.cascade(CascadingAction.java:47) at org.hibernate.engine.Cascade.cascadeToOne(Cascade.java:213) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:157) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:108) at org.hibernate.engine.Cascade.cascadeCollectionElements(Cascade.java:290) at org.hibernate.engine.Cascade.cascadeCollection(Cascade.java:185) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:160) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:108) at org.hibernate.engine.Cascade.cascade(Cascade.java:248) at org.hibernate.engine.Cascade.cascade(Cascade.java:223) at org.hibernate.event.def.DefaultDeleteEventListener.cascadeBeforeDelete(DefaultDeleteEventListener.java:220) at org.hibernate.event.def.DefaultDeleteEventListener.deleteEntity(DefaultDeleteEventListener.java:169) at org.hibernate.event.def.DefaultDeleteEventListener.onDelete(DefaultDeleteEventListener.java:110) at org.hibernate.impl.SessionImpl.fireDelete(SessionImpl.java:712) at org.hibernate.impl.SessionImpl.delete(SessionImpl.java:706) at org.hibernate.engine.CascadingAction$2.cascade(CascadingAction.java:47) at org.hibernate.engine.Cascade.cascadeToOne(Cascade.java:213) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:157) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:108) at org.hibernate.engine.Cascade.cascadeCollectionElements(Cascade.java:290) at org.hibernate.engine.Cascade.cascadeCollection(Cascade.java:185) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:160) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:108) at org.hibernate.engine.Cascade.cascade(Cascade.java:248) at org.hibernate.engine.Cascade.cascade(Cascade.java:223) at org.hibernate.event.def.DefaultDeleteEventListener.cascadeBeforeDelete(DefaultDeleteEventListener.java:220) at org.hibernate.event.def.DefaultDeleteEventListener.deleteEntity(DefaultDeleteEventListener.java:169) at org.hibernate.event.def.DefaultDeleteEventListener.onDelete(DefaultDeleteEventListener.java:110) at org.hibernate.impl.SessionImpl.fireDelete(SessionImpl.java:712) at org.hibernate.impl.SessionImpl.delete(SessionImpl.java:692) at com.xxx.xxx.factory.Factory.makeTransient(Factory.java:xx) at com.xxx.xxx.activities.Level1Activities.deleteLevel1(Level1Activities.java:xx) where my mapping file looks something like <hibernate-mapping> <class name="com.xx.xxx.model.Level1"> ... <map name="metadata" cascade="all-delete-orphan" lazy="extra"> <key column="LEVEL1_ID"/> <index type="java.lang.String"> <column name="KEYY" not-null="true"/> </index> <one-to-many class="com.xxx.xxx.model.Level2"/> </map> </class> <class name="com.xxx.xxx.model.Level2"> ... <many-to-one class="com.xxx.xxx.model.Level1" name="owner"/> <set name="tokens" cascade="all-delete-orphan" inverse="true" lazy="extra"> <key column="LEVEL2_ID"/> <one-to-many class="com.xxx.xxx.model.Level3"/> </set> </class> <class name="com.xxx.xxx.model.Level3" lazy="true"> ... <many-to-one class="com.xxx.xxx.model.Level2" name="owner" insert="false" update="false" not-null="true" access="field"/> <many-to-one class="com.xxx.xxx.model.Level4" name="term" insert="false" update="false" not-null="true" access="field"/> </class> <class name="com.xxx.xxx.model.Level4"> ... <property name="data"/> </class> </hibernate-mapping> In order to trigger the exception, I have to create Level1 objects with level4 great-grandchildren 512 times. This number varies between different cases, seemingly arbitrarily, but it is deterministic. Or, rather, the exception doesn't happen until I add a certain large number of children, grandchildren, and great-grandchildren. However, it always happens at the same point.
FWIW - I ran into this same issue on JDK 1.5/Linux/client mode, while never seeing it under Windows:
java version "1.5.0_03" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_03-b07) Java HotSpot(TM) Client VM (build 1.5.0_03-b07, mixed mode, sharing) And it appears to have been fixed by running in server mode, though only time will tell since it seems to appear sporadically Whoops, replace Term -> Level4 on the stack trace.
Hi y'all. Maybe this helps some of you: I have tried the sdk 5.0 from IBM, both 32-Bit and 64-Bit on Linux. Both versions work fine.
You'll notice the versions are beta right now. Follow this link: https://www14.software.ibm.com/webapp/iwm/web/reg/pick.do?source=lxdk&lang=de_DE (Sorry, German version ;-) ). You'll have to register for free. I saw the similar error. It does not happen at the very beginning. The exception is thrown out after the query run about thousand times.
It is on linux, java1.5. If the same code running on Windows, it never throws the exception. java.lang.NoSuchMethodError: com.brightcove.reporting.records.VideoStreamedDetailRecord.getHibernateLazyInitializer()Lorg/hibernate/proxy/LazyInitializer; at com.brightcove.reporting.records.VideoStreamedDetailRecord$$EnhancerByCGLIB$$d869ed93.getHibernateLazyInitializer(<generated>) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:274) at org.hibernate.type.EntityType.resolve(EntityType.java:303) at org.hibernate.type.EntityType.nullSafeGet(EntityType.java:217) at org.hibernate.impl.IteratorImpl.postNext(IteratorImpl.java:93) at org.hibernate.impl.IteratorImpl.next(IteratorImpl.java:120) at com.brightcove.payments.PaymentSchedulerFacade.processRecordForCorporateAccount(PaymentSchedulerFacade.java:404) at com.brightcove.payments.PaymentSchedulerFacade.processVideoStreamedDetailRecords(PaymentSchedulerFacade.java:144) at com.brightcove.payments.ProcessUsageDataJob.executeJob(ProcessUsageDataJob.java:24) at com.brightcove.scheduler.SchedulerJob.execute(SchedulerJob.java:22) at org.quartz.core.JobRunShell.run(JobRunShell.java:191) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516) Just wanted to say, that I see the same error. Strangely I don't see it on my development machine, but only on the production one. Both are linux and have the same distribution.
They should also be using the very same java version. I will try to find all differences and report back. Is it possible to reproduce this problem in single thread ?
After we added lazy="false" to that mapping, it seems the problem is gone.
Precisely to say, last time, the problem happened after we processed 2000 records. Now, the code successfully processed 200,000 records. I am not sure how many query are executed. Anyway, we are doing more testing. lazy="false" effectively disable the part of the code that uses the not-found method.
Can anyone tell me if this only happen if you are accessing the same object in two different threads at the same time ? session is of course not threadsafe, but invocation of proxied methods should be, correct ? Proxied method is not thread safe if interceptor is not thread safe (lazy loader is not thread safe), but it looks like a problem specific to java/linux versions.
Add Linux kernel version to report ( linux kernel 2.4.20 and "better" must be used with java 1.5 ). Juozas: I am using 2.4.27 (Debian Sarge), and the problem appeared with
java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing) For what it's worth I have the problem too using Debian Sarge, kernel 2.6.11.
I have it with Sun Java 1.5.0_04, and it doesn't go away if I run it with the -server flag. I have also tried it with the IBM 5.0 JRE, exactly the same problem. This is a build server running CruiseControl (single threaded as far as I know), tests run okay and then at some point (not always the same) they start failing and then every test fails. Tests all run fine on Windows. Server VM solves the problem for us with:
java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode) Can somebody report that server VM 1.5.0_06 also fails? It is possible to copy hibernate 3.05 code and to maintain local hibernate copy if this very urgent (there is nothing wrong in this code, it was just optimized for startup and "finalize" method was removed from proxies). But I am afraid it can be more problems http://forum.hibernate.org/viewtopic.php?t=951276&highlight=identityhashmap There is no information about linux and jvm versions, but it looks like the same JVM memory issue too. it looks like 1.5 JVM is unstable on linux kernel 2.6.11 too,
BEA recommends 2.6.12 ( I am not sure it is tested on this version ) . Ok, i'm now upgrading this to a Blocker since its getting more and more obvious that the optimization that was done in 3.1.x to not include finalizers() is causing issues on certain VM/os-combinations.
We need to a) verify who is to "blame", is the code we have wrong or is it because of bugs in the vms ?(last time we brought that issue up it was stated it should be safe according to spec - so we we wrong or does the vm not implement the spec ?) b) what is the *exact* patch that we can provide here in the shortterm so it will come back to the old much less efficient (but more resistant) code No patch is need, we just need to test BasicLazyInitializer 1.13 and CGLIBLAazyInitializwer 1.14
(HHH -1103). It must be tested in single thread with different JVM versions to "blame" JVM+Linux, I do not have any linux server and "bad" code to test it myself. We have kernel 2.6.14 on Fedora Core 4 and the problem is there with client VM
If you guys need a full application that will repeat the error, the source code is here:
http://cvs.photostructure.com I built a web archive for your installing convenience here: http://home.comcast.net/~mrmceachen/photostructure.war (which was just built off v0-7-3454) I saw the error on Linux 2.6.8-2-386 (Debian/testing), and can verify that the problem has (so far) gone away when I added -server to the JVM (running in an apache tomcat 5.5.15 servlet container): java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode) (I know this isn't a stripped-down simple test case that repeats the issue, but at least it's something...) The error showed up for me initially after a couple hundred images were harvested, then very quickly after JVM bounce thereafter. Email me if you want more details or if this is helpful. Exception stacktrace for the code submitted above:
java.lang.NoSuchMethodError: com.photostructure.model.Label.getHibernateLazyInitializer()Lorg/hibernate/proxy/LazyInitializer; com.photostructure.model.Label$$EnhancerByCGLIB$$5f710a32.getHibernateLazyInitializer(<generated>) org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:274) org.hibernate.type.EntityType.resolve(EntityType.java:303) org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842) org.hibernate.loader.Loader.doQuery(Loader.java:717) org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224) org.hibernate.loader.Loader.loadCollection(Loader.java:1919) org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:71) org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:520) org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:60) org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1627) org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:344) org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86) org.hibernate.collection.PersistentSet.iterator(PersistentSet.java:138) com.photostructure.model.AssetImpl.getLabelLineages(AssetImpl.java:157) com.photostructure.model.AssetImpl$$FastClassByCGLIB$$e74f262b.invoke(<generated>) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) org.hibernate.proxy.CGLIBLazyInitializer.intercept(CGLIBLazyInitializer.java:161) com.photostructure.model.JpegAssetImpl$$EnhancerByCGLIB$$84fefee6.getLabelLineages(<generated>) com.photostructure.web.ViewController.handleRequest(ViewController.java:146) org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:717) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:658) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:347) javax.servlet.http.HttpServlet.service(HttpServlet.java:689) javax.servlet.http.HttpServlet.service(HttpServlet.java:802) org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) Try -XX:+UseTrainGC, this garbage collector was not changed since 1.4.2 and if it helps then probably "finalize" confuses JVM (default garbage collector implementation on linux).
This option does not affect the error. Nothing changes.
I don't think GC can cause java.lang.NoSuchMethodError
> I don't think GC can cause java.lang.NoSuchMethodError
It can if it garbage collects a class, and the classloader then needs to fetch the bytecodes again. To prove that hypothesis, use a client JIT with "-Xnoclassgc". If the problem doesn't show up, that's the issue. I wanted to try it also but so far haven't had time.
What makes you think that the code is treated the same way as data in JVM and that it gets garbage collected as regular data? "-Xnoclassgc" doesn't help. Still get the same exception in roughly the same place.
"client/server" switch affects not only GC but also compilers. I believe, but I may be wrong that the client VM uses a combination of interpreter and compiler while server VM uses only compiler with much more expensive optimizations. I think the problem may be an indication of a bug in either interpreter or client compiler or the client code responsible for method recompilation from interpreted code to compiled code when the method is detected to be "hot". This of course assumes that the bug is not caused by cglib, which I don't know for sure. java.lang.NoSuchMethodError is a side effect, interceptor instance is lost in "resolveIdentifier" after it was set in "getProxy()". Generated byte code is same for JVM and OS versions (it uses JAVA 1.2 compatable class format).
Generated Proxies are same on hibernate 3.05 and 3.1, but 3.1 proxies are without "finalize" method. I see it depends on JVM and linux, but it is hard to believe, JVM or OS bugs cause segmentation faults, core dumps or deadlocks. I suggest to build hibernate and to test it with "finalize" method. Oh yes. Here you are right. I forgot the fact that the problem doesn't exist on Windows, which rules out a bug on Java level.
> What makes you think that the code is treated the same way as data in
> JVM and that it gets garbage collected as regular data? Oh, I wasn't saying that data and class bytecodes get GC'ed in the same manor--I don't know low-level implementation details of the client VM--but I have seen the client VM GC classes that haven't been used for a while (and I would guess it's more aggressive if the heap is already at max). You can see it happen with -Xloggc. I don't know what exactly the "finalize method" entails--if you guys give me a .patch file to apply to the hibernate 3.1 tree (or just a .tgz with .java files to dump into src), I'd be happy to try it out. (Or is the code change to bcprov? In any event, I've just verified that I can reproduce the issue in < 10 minutes).
1. I don't believe that class GC has something to do with generated code. I believe that generated code is not stored in Java heap and thus cannot be garbage collected. It is stored in a separate area of memory that is not heap and not managed by GC.
2. I don't seem to be able to reproduce the problem while using "-Xint" switch (interpreted mode execution only), which again points us to the possible bug in client VM compiler. Can anybody try "-Xint" switch and see what happens? Of course your app will run several times slower, but it is worth it. FWIW, I haven't seen the issue yet with a '-Xint' JVM.
This issue also doesn't seem to effect latest Java 6.0 snapshot: jdk-6-beta-bin-b68-linux-i586-19_jan_2006.bin
I am also seeing this error in my envrionment. If it will help, here are the specfiics of what I am seeing and how I duplicate the error.
Linux 2.6.11 MySQL 4.1.x Java 1.5.0_03 (Sun JVM) Hibernate 3.1.1 Tomcat 5.0.28 Dual PIII When I run with the '-server' flag, I do not get the error (at least not yet or with a test case). To trigger the error, I have 1 class (Person) with a bunch of Set relationships. For each iteration I load the primary object and then each of the relationships. In my setup, this loop triggers the error between 50 and 60 iterations. There error is thrown on the log.fatal line. Here is the error: Exception in thread "main" java.lang.NoSuchMethodError: com.itsolut.entity.survey.usr.UserTest.getHibernateLazyInitializer()Lorg/hibernate/proxy/LazyInitializer; at com.itsolut.entity.survey.usr.UserTest$$EnhancerByCGLIB$$559189a0.getHibernateLazyInitializer(<generated>) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:274) at org.hibernate.type.EntityType.resolve(EntityType.java:303) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842) at org.hibernate.loader.Loader.doQuery(Loader.java:717) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224) at org.hibernate.loader.Loader.loadCollection(Loader.java:1919) at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:71) at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:520) at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:60) at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1693) at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:344) at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86) at org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:109) at org.hibernate.collection.PersistentSet.size(PersistentSet.java:114) at com.itsolut.backoffice.config.MaintDB.triggerError(MaintDB.java:80) at com.itsolut.backoffice.config.MaintDB.main(MaintDB.java:38) I first get a list of primary IDs and then load each object (and relationships) in a new Session. private static void triggerError() throws Exception{ Configuration configuration = new Configuration(); configuration.configure("/com/itsolut/entity/hibernate.cfg.xml"); SessionFactory factory = configuration.buildSessionFactory(); Session session = factory.openSession(); Statement st = session.connection().createStatement(); ResultSet rs = st.executeQuery("select partyID from bo_party where companyID=11 and type='P'");// limit 0,100"); List<Long> ids = new ArrayList<Long>(); while(rs.next()){ long id = rs.getLong(1); ids.add(new Long(id)); } st.close(); session.close(); int index=0; for(Long id: ids){ session = factory.openSession(); Person person = (Person)session.load(Person.class, id); log.fatal("count["+(index++)+"] " + " person id["+person.getPartyID()+"] fname["+person.getFname()+"]" + " org id["+person.getOrganization().getPartyID()+"] name["+person.getOrganization().getName()+"] " + " titles["+person.getUserTitles().size()+"] " + " notes["+person.getNotes().size()+"] " + " activities["+person.getActivities().size()+"] " + " addresses["+person.getAddresses().size()+"] " + " paymethods["+person.getMethods().size()+"] " + " opportunities["+person.getOpportunities().size()+"]" ); session.close(); } } so you can reproduce this in a *single* thread ?
So it looks like a bug in JIT compiler (try -Xint) and it must be better to report it for SUN than to move "finalize" back, it is a good optimization and JIT can fail with old code anyway..
Your report has been assigned an internal review ID of: 638664
dateCreated: Mon Jan 30 12:06:47 MST 2006 type: bug jdcid: kyrylkov status: Waiting category: java subcategory: runtime release: 5.0 hardware: x86 OSversion: linux priority: 4 synopsis: java.lang.NoSuchMethodError description: FULL PRODUCT VERSION : java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing) ADDITIONAL OS VERSION INFORMATION : Linux kernel 2.6.x EXTRA RELEVANT SYSTEM CONFIGURATION : client VM <Max> Yes I was able to reproduce this error in a single thread. <Juozas> I'll try the -Xint option later to see if the issue is still present; however, I'm not a big fan of running my application in interpreted-only mode. <Sergiy> Is there a link you can post/share so that others can follow the dialog on this bug with Sun. Thanks, Chris.... it looks like the -Xint option also works for my environment. I let the process run until I manually killed it. It queried roughly a thousand records (it was hitting the error at 54 records without the switch).
If there is a workaround you would like to try, let me know and I can try it out. Chris.... The bug is in internal review phase. It means it is not accepted/published yet. As soon as I get update from Sun I will post it here.
So has anyone been able to try the 2.6.12 kernel to verify whether or not the problem shows up there? If this is something limited to a specific kernel version of a specific OS, I am inclined to call this 'not a bug'
I have kernel 2.6.13, sun JDK 1.4.2_04-b5, glibc with NPTL, and the problem still is.
Seems to work with -server, but I did not test very much. I would very much call it a bug until somebody figures out exactly what's happening (I want to migrate an application from hibernate 2.1.8 to 3.1.2, but this bug is really blocking me - I mean, who knows what else may break from the same cause, and this migration is not easily reversible). "but it
takes "several invocations"/"some time" until it appear, but then it appears very often" Probably it happens after JIT compiles "hotspots". Funny, considering so far this seems to only occur on certain Linux kernels I would very much call it not a bug. Not a bug in Hibernate/CGLIB at any rate; more likely a bug in the JVM implementations for those Linux builds.
So just to cover all bases and rule out CGLIB-Hibernate interaction, would someone experiencing this issue be willing to check out CVS HEAD and try out the new Javassist support?
I'd agree with the outline info, it doesn't appear to happen right away but after a while of use.
Versions here: Multi-CPU SMP Intel x86 machine Linux 2.6.12.2 Kernel (libc-2.3.4, NPTL) Sun JVM build 1.5.0_06-b05, mixed mode Hibernate 3.1.2 (with cglib 2.1.3, commons-collections-3.1, ...) Tomcat 5.5.15 I think the mode the JVM runs in the default one, the application runs within tomcat. Application Outline: There are multiple threads in play, each thread has its own Session to work with. These threads are fairly long running, they run for ~15 mins, only do 5 seconds of work in that time. Each thread gets its own Session instance soon after it starts and will keep it until the thread terminates. Transaction state is managed inside the relevant worker method, most of the wall clock time is spent with an open Session but no active transaction context and no uncommitted dirty data. hibernate.transaction.auto_close_session=false I dont get any NoMethodException within 3 to 4 minutes of firing up the application, even through within 30 seconds everything is running and hot and the same code path that throw out exceptions later have been executed at least a few times. If its a hotspot issue, how does hotspot decide to do its work. Does adding a simple Thread.sleep() delay to slow down your reproducable application after the point when the problem occurs. I really dont believe my code gets many iterations before I get the error, but what does happen is a lot of waiting and delays between iterations. http://opensource2.atlassian.com/projects/hibernate/browse/HHH-1293#action_21914 by Chris Hane <== is that a working Single Thread example ? Is it possible to snapshot the VM with $JAVA_HOME/bin/* tools to work out whats -Xint, client, server at any given moment ? I'm just auditing my app code, to make sure it is not possible for one thread to hand another thread a proxied object, and then the source thread shutdown (causing a Session.close()) I dont think this is the case for me. I see no way to fix this issue without hacks like fake "finalize" and I sugest to documment this issue with workarounds "-int or -server" in release notes.
"-server" is not a workaround - it just makes hotspot optimize the code later (i.e. greater number of invocations), that's why I think it appears to be working.
And I do not think disabling hotspot is really an option for production environments. So is there any real workaround for this problem? Is it a JDK for linux that actually works? So "-server" doe's not help for you and it cuses this error anyway ?
client and server "hotspot" are two different compiler implementations http://java.sun.com/products/hotspot/docs/whitepaper/Java_HotSpot_WP_Final_4_30_01.html#pgfId=1082363 if both compilers have this bug then "-Xint" is a single workaround. I think it does not make sence to adapt code for JIT, it looks like same problem exists with hiberntae "IdentityHasMap" and probably in many places, code will be affected by this bug and application will not be reliable on this platform anyway.
Just to say my code has been working fine with tomcat started up in "-server" mode rather than "-client" the default that jsvc was picking up.
I was under the impression "-server" make the JVM optimize the code SOONER as in it always runs compiles into optimized form first then runs it, there is no interpreted code run, nor any mixed, therefore there is no hotspot working (hotspot I've understand to be an optimization technique which decides when to turn -Xint into machine code based on runtime statistics working out the most executed code, a trade off between startup time and execution time, I believe hotspot is used with -client that support mixedmode but not with -server, since everything is optimized before execution). With JVM 1.5.0_06 I understand things to be: -Xint forced interpreted mode only, disable hotspot and disable optimization -client starts off with interpreted mode, and hotpost decides if/when a block of code is to migrate to optimized form -server always takes bytecode and optimizes before execution I guess the bug is really as Steve suggests, in the procedure the JVM uses to migrate with hotspot to fully a optimized unit of code and then migrate execution within the application from -Xint to the optimized code. If the problem does not turn up with -Xint nor with -server then this code migration technique process has to be the problem. Some evidence to back this up:
-client (BAD, app runs for 4 mins before first NoSuchMethodError) -server (GOOD, app has run for 30 hours without any problem) -server -XX:CompileThreshold=100 (GOOD, app has run for 60 minutes without any problem) -client -XX:CompileThreshold=100 (BAD, app runs for 30 seconds before first NoSuchMethodError) -client -XX:CompileThreshold=3000 (BAD, app runs for 18 minutes before first NoSuchMethodError) -Xint (GOOD, app has run for 30 minutes without any problem) I can not find any documentation to back up my claim that -server immediatly executes optimized code, I can find reference that implies with -server a more optimizing compiler is used (than -client) and hotspot kicks in after more iterations. Useful link for SUN JVM: http://java.sun.com/docs/hotspot/VMOptions.html I've made a mistake in my previous post - assumed that -server would fail because hotspot will optimize after a greater number of invocations (although it is true it will optimize after a greater number of invocation, see hotspot VM options).
I tested now with my configuration (kernel 2.6.13, libc with nptl, sun jdk 1.4.2_04-b05), and I confirm that it works fine with "-server". So, disregard what I've said in the previous comment, "-server" is a workaround from my point of view. I've just tested the hibernate CVS head with javassist bytecode provider.
It works fine with client VM. Also, it still fails with cglib. With the server VM there is still recompilation happening and there is a mechanism to recompile hot methods. First a method is compiled with the cheapest optimizations, later if it is determined to be hot it is recompiled with more expensive optimizations. This actually can happen several times, depending on the number of optimization levels available in the compiler.
So what is different with the client VM? First we have interpreter to compiler transition, which may be one place where the JVM bug exists. Second even the client HotSpot may have several levels of optimizations, so a method can be further optimized later with more expensive optimizations. Since Sun says that client and server HotSpots are totally different implementations, the bug may also exist in recompilation stage of the client HotSpot compiler. Probably all these things can be tracked with appropriate JVM logging turned on. We just hit this problem at our company (Java 1.4.2_10, RHEL 3.0ES, Hibernate 3.1.2), and I'd like to verify that turning on -server will permanently fix the problem, since we intend to use Hibernate 3.1.2 in a shipping product.
I also have a couple of comments. I'm skeptical about whether or not one of the suggested fixes above: - en.setCallbackTypes( CALLBACK_TYPES ); - en.setCallbackFilter( FINALIZE_FILTER ); + en.setCallbackType( MethodInterceptor.class ); really solves the underlying problem. However, if that does work, would it be okay to make that change, and put back the following lines of code in BasicLazyInitializer to compensate: else if ( "finalize".equals(methodName) ) { return null; } ? Also, it seems likely to me that what really triggered this is the change in CGLIBLazyInitializer that replaced this: final HibernateProxy proxy = (HibernateProxy) factory.newInstance(instance); with this: final HibernateProxy proxy; try { proxy = (HibernateProxy) factory.newInstance(); } catch (Exception e) { throw new HibernateException( "CGLIB Enhancement failed: " + persistentClass.getName(), e ); } ( (Factory) proxy ).setCallback( 0, instance ); Even though interceptDuringConstruction is false, I'd feel a lot better about things if the MethodInterceptor instance is set when the proxy is constructed, rather than afterward where the optimizer can inline, rearrange, and do whatever it is doing right now. Would it be possible to go back to using the old way of constructing proxies (using Factory.newInstance(interceptor)), and would this fix the problem, even with a slightly broken optimizer? I can tell you anecdotally that we had the problem, switched to -server which immediately fixed it, and our app has been going through acceptance testing for a few weeks without further issues.
I can confirm that '-server' has fixed the problem in my app, which has been stress tested for four weeks. Sun Java 1.4.2_09, RHEL 3.0ES, Hibernate 3.1.2
Greetings,
i have just discovered this problem and i would like to add to the above conversation that i used the 3.1 rc2 version with no problems. But when i changed to 3.1.2 then i got it. Since the test was made on the same Linux system (2.6.9-22.0.2.EL) , java 1.4.2._09 , tomcat 5.0.28 i have to ask whether the problem is at the code of hibernate after the rc2 version. How could the rc2 version work and not the new one. There must be a problem at the changes. I hope that someone reads this message and that the topic is not dead. Thank you very much Leonidas P.S> please excuse me if my hypothesis is wrong... Leonidas,
It is not a problem of Hibernate. It looks like it is a change in cglib that exposed a bug in Sun's JVM. -server option is the solution that works. You may try an older cglib, but I don't know whether it will work or whether there are some dependencies that can break.
There _was_ a change in Hibernate, which is the one I mentioned earlier as probably having triggered this: Rather than creating the CGLIB proxy instance using a factory, which sets the interceptor reference during construction, the Hibernate code was changed to create the CGLIB proxy using Class.newInstance, and then set the interceptor _after_ the proxy is constructed.
Since the symptoms of this bug indicate that the optimizer is causing the interceptor to not get set into the proxy, I think it is very likely that changing the way that Hibernate constructs the proxies is what caused this optimizer problem to be exposed, and not any change to the CGLIB. I am all in favor of going back to using the CGLIB Factory interface to create the proxies with the interceptor set in the constructor, the way the Hibernate code used to do it, even though without optimizer bugs, the two methods should be equivalent. (The Factory way requires constructing a dummy instance of the proxy class, which may be why they changed the code?) But, I've already said this in my previous post. So, twice is enough. Factory interface forces to initialize proxy class on startup, <clinit> is very heavy. Probably good workaround is to try "InvocationHandler" callback type instead of "MethodInterceptor", it will generate light weight code. Try to experiment with this stuff and patch hibernate if you can reproduce this problem and understand cglib API.
OK guys, I have created repro case for a case that I think is highly related to the case.
It fails on Linux without -server option and passes when the option is present. Note, the phenomena is NOT present in 64bit version of JVM So the environment: kosta@localhost /usr/java/lib/hibernate-3.1.1 $ java -version java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing) kosta@localhost /usr/java/lib/hibernate-3.1.1 $ uname -a Linux localhost 2.6.14-gentoo-r2 #1 Mon Nov 28 00:04:16 Local time zone must be set--see zic manu x86_64 AMD Athlon(tm) 64 Processor 3500+ AuthenticAMD GNU/Linux The repro case is for Hibernate testing environment, I just made simple modification to help to run individual test cases, to run the case simply unarchive the file into existing hibernate-3.1.1 directory. Invoke ant -f build-test.xml junit -Dtestpattern=ManyS*Test and it should fail on 32bit JDK if you will now uncomment <jvmarg value="-server"/> option in the build-test.xml file the test passes without problems. Odd, without -server option on 32bit JVM test always fails on 1500th pass. Testcase: testManySessions took 1.505 sec FAILED 1499-SystemParameter.getSysParamName() = null junit.framework.AssertionFailedError: 1499-SystemParameter.getSysParamName() = null at org.hibernate.test.manysessions.ManySessionsTest.testManySessions(ManySessionsTest.java:35) at org.hibernate.test.TestCase.runTest(TestCase.java:150) We're getting the same type of exception on our Linux system. We were able to get around the error by passing the -server argument to the VM. We were not able to reproduce the error on our Windows development systems.
$ uname -a Linux hostname 2.4.21-32.0.1.ELsmp #1 SMP Tue May 17 17:52:23 EDT 2005 i686 i686 i386 GNU/Linux $ /usr/java/j2sdk/bin/java -version java version "1.4.2_04" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_04-b05) Java HotSpot(TM) Client VM (build 1.4.2_04-b05, mixed mode) $ /usr/local/tomcat/bin/catalina.sh version Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JAVA_HOME: /usr/java/j2sdk Version: Apache Tomcat/5.0.28 $ cat hibernate.cfg.xml <hibernate-configuration> <session-factory> <property name="dialect">org.hibernate.dialect.SQLServerDialect</property> <property name="show_sql">false</property> <property name="use_outer_join">true</property> <property name="connection.username">***</property> <property name="connection.password">***</property> <property name="connection.driver_class">net.sourceforge.jtds.jdbc.Driver</property> <property name="connection.url">jdbc:jtds:sqlserver://localhost:1433/database</property> <property name="use_sql_comments">true</property> <property name="connection.autocommit">true</property> <property name="cache.provider_class">org.hibernate.cache.EhCacheProvider</property> <property name="cache.use_second_level_cache">true</property> <property name="c3p0.min_size">5</property> <property name="c3p0.max_size">100</property> <property name="c3p0.max_statements">50</property> <property name="c3p0.max_statements">1800</property> <!-- Mappings ... --> </session-factory> </hibernate-configuration> $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.20GHz stepping : 1 cpu MHz : 3192.116 cache size : 1024 KB ...snip... processor : 1 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) CPU 3.20GHz stepping : 1 cpu MHz : 3192.116 cache size : 1024 KB ...snip... $ free total used free shared buffers cached Mem: 2055296 1857272 198024 0 237396 780924 -/+ buffers/cache: 838952 1216344 Swap: 2096440 73448 2022992 $ ps aux --width=2048 |grep tomcat user 20619 26.7 20.1 807072 414688 ? S 11:47 5:22 /usr/java/j2sdk/bin/java -Xms128m -Xmx512m -verbose:gc -server -Djava.endorsed.dirs=/usr/local/tomcat/common/endorsed -classpath /usr/java/j2sdk/lib/tools.jar:/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap s Using the jdk 6 beta has solved this problem for me.
kosta@swan ~ $ java -version
1.6-beta is by default uses 'server' mode java version "1.6.0-beta" Java(TM) 2 Runtime Environment, Standard Edition (build 1.6.0-beta-b59g) Java HotSpot(TM) Server VM (build 1.6.0-beta-b59g, mixed mode) Using the following setup:
- Apache 2.0.53 - Resin 3.0.18 - Hibernate 3.0.12 - MySQL 5.0.15 For anyone using Resin, I solved this issue using the JVM -server option as well. For the startup script, just use the argument '-J-server' to pass in arguments to the JVM: su $USER -c "/usr/local/resin/bin/httpd.sh -verbose -J-server -server-root $ROOT -conf $ROOT/resin.conf start Default can be set in jvm.cfg file, it is in jre/lib/i386 directory (location can depend on jvm version).
There can be problems when using older jsvc versions for tomcat and the server vm mode (jsvc couldn't find the server vm). Can be fixed by compiling the newest commons-daemon 1.0.1 jsvc version against a 1.5 JDK.
The problem was fixed for me after the upgrade of jsvc and using the "-jvm server" switch for startup of tomcat. Yep!
It seems -server works! I found the problem already present in hibernate 3.2 rc2 without -server parameter.
Environment: RedHat Enterprise Linux AS 4 Tomcat 5.5.17 SUN JDK 1.5.0_06 The same problem exists with Mac OS X's jdk1.5_06 on intel. The same -server workaround works. I'd guess that means that Apple is basing the intel JVM off of the linux source base.
This problem exists in our environment. We are running jdk1.5.0_06 on Fedora Core 5, kernel 2.6.16-1.2122, Hibernate 3.1, CGLIB 2.1.3. The problem happens with the client VM, and disappears when the -server parameter is used. The application where we experienced this problem is a single-threaded app that is using Hibernate to run many large selects.
We are using Hibernate 3.1.3 on Linux 2.6.13-1.1532_FC4 with java:
java version "1.4.2_10" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_10-b03) Java HotSpot(TM) Server VM (build 1.4.2_10-b03, mixed mode) And neither -server nor -Xint are working for us since upgrading to 3.1.3. -server with Hibernate 3.1 was working with this setup. Is anyone else running 3.1.3 successfully? Why does this bug have twice as many votes as the next closest bug, is marked as a Blocker and yet has not been fixed in six months? Rollback the original code change already! Because this bug is a *JVM bug*, complains to your vendor or use another one which will work just fine.
@ sergiy:
Is there any news regarding the bug report at SUN. I know it has already been asked, but could you post a link? Andreas ,
I haven't heard back from Sun since I submitted the bug. My unit tests were forking a VM without the -server flag. Sorry for the false alarm. It is now working for us.
@Emmanuel: just because Hibernate is not the root cause does not mean the Hibernate guys should walk away here. This was working in 3.1rc2 and then broke in 3.1 final. Look at the number of comments and votes on this issue - it's causing a lot of users pain and so is a prime candidate for a workaround. I would like to understand why the Hibernate team can't roll back the change which caused the problem in the first place. @mike: while I sympathize with you, the change in Hibernate code did not _cause_ the problem, it only triggered a JVM bug. We should be pestering Sun about this, not the Hibernate team.
Meanwhile, try using BEA's JRockit VM. so use 3.2 and javassist
Client JVM on linux will fail anyway, probably it will fail later, but there are same problems in many places. Probably it is possible to find random workarounts on client JVM, but I am afraid we can break more things this way. I think "-server" is not so bad workaround.
Currently I'm working with Sun Java 1.6.0 Beta. The client VM does not seem to crash.
The problem with the server VM is it's coming up quiet slowly, so sometimes coding and testing is a pain in the unmentionable. Maybe using 1.6.0 Beta will help some of you. In a productive environment the server VM cannot be a problem, so maybe this is some kind of workaround. Attaching a custom build of Hibernate. This is 3.1.3 + rollback of
why did you attach a binary jar ? a patch would be much more sensible.
btw. Max, the patch is basically just reversing the pluses and minuses in the patch attached to
I'm sure the fix is important but it's more important to us that our app actually work. For very specific reasons we cannot use the workarounds specified here so the binary is just another option for people if they are in the same situation. Hi, I haven't found the root cause of the failure (I'm still trying to understand that), however, I do have a patch that seems to help both
org.hibernate.test.proxy.ProxyTest works correctly with the hack patch (tested on Linux). The org.hibernate.test.manysessions.ManySessionsTest works until we run out of memory (this seems to occur on Windows + Linux). I'm not sure but I suspect that the out of memory error is separate as my standalone version of ManySessions doesn't run out of memory and it does 10000 iterations instead of just 5000. The patch changes "org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.public static HibernateProxy getProxy( final Class factory, final String entityName, final Class persistentClass, final Class[] interfaces, final Method getIdentifierMethod, final Method setIdentifierMethod, final AbstractComponentType componentIdType, final Serializable id, final SessionImplementor session)" to ignore the factory parameter and instead invoke the other getProxy method that doesn't take the factory. I'm new to a lot of this code and am still trying to understand the difference between newing up the Enhancer each time versus using the factory. This is not the complete solution yet as I don't understand why this helps yet (e.g. what is wrong with using the factory).
I am attaching a cleaner patch. I re-read the above comments (especially what Richard J. Smith said). In the new patch, we treat the factory similar to org.hibernate.bytecode.cglib.ProxyFactoryFactoryImpl.factory.
I would like to make one more change still for this bug. Please try the updated CGLIBLazyInitializer.patch. The attached patch looks good, I fixed the outofmemory error by specifying max memory of 256m for junitsingle case.
The only improvement that I would make is to: 1. Change org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.factory member variable to be of type net.sf.cglib.proxy.Factory (or just type Object if we don't want to change method signatures). 2. Change org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxyFactory(Class persistentClass, Class[] interfaces) to return Factory or Object (depending on whether we want to change the signature and also the javassist calls). 3. Change org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.HibernateProxy getProxy( final Class factory, final String entityName, final Class persistentClass, final Class[] interfaces, final Method getIdentifierMethod, final Method setIdentifierMethod, final AbstractComponentType componentIdType, final Serializable id, final SessionImplementor session) To pass 'factory' parameter as a Factory or Object type. This would reduce the number of objects that we NEW up during lazy fetching. hi scott :)
Your patch is nice and simple and I think that is at least worth a shot. Unfortunatly i don't have a linux machine/jdk to verify this against. I'll try and get my hands on one tomorrow. Are you using the manysessions.tgz test to verify it ? just got word from Steve ;)
great that someone with a linux box could get around fixing this. one thing though with the change in signature - are you completely sure that will guarantee to work with respect to classloaders etc. ? hi max :)
Yes, I used a combination of manysessions.tgz to verify this problem and also the existing org.hibernate.proxy.ProxyTest test (both of them have to pass). The manysessions test failed everytime I ran it prior to the fix and it worked every time I ran it after the patch was applied. The signature change would have to be from type Class to Object as net.sf.cglib.proxy.Factory is specific to cglib.
So the ProxyFactory method will return Object instead of Class and the getProxy will accept it as an Object. In the case of cglib, we will return the factory object. In the case of javassist, we will continue to return the class object. The same will be true for cglib getProxy, we will pass the factory object and javassist will continue to pass the class object. The only thing we will lose is type safety. I could avoid making this change at the cost of additional object allocation in cglib getProxy method. If anyone was calling getProxy() or getProxyFactory() from other locations (perhaps dynamically since I didn't see any other calls), I would break them with this change. Is this what you mean with regard to classloaders? Or are you referring to my original (bad) idea of passing type net.sf.cglib.proxy.Factory? My concern is just that we have previously had issues with what classloaders are/will be used at certain times.
And passing the same object instance around could change that (e.g. before the new were always done in context of the callee) where as now it will be done at "setup time". This can have consequences in different containers....but in proper isolated containers it should work fine. Why only change from Class to an instance in the cglib scenario and not also in javaassist ? (aren't they symmetrical in this regard?) With regard to classloader, it will help me think about that by laying out the steps of the current code and the patched code.
With respect to the cglib support, we currently return a class from getProxyFactory() and create an instance from that class inside of getProxy() via the newInstance static class method. The class definition is loaded inside of getProxyFactory() and used within getProxy(). With respect to the proposed patch, we will load the factory class in getProxyFactory() and also create an instance from that class inside of getProxyFactory(). We will return the Factory instance from getProxyFactory(), such that all class loading should occur within getProxyFactory(). In getProxy(), we will use the passed Factory instance to create an instance of the proxy class. To answer the raised point about classloader, the question that I need to investigate is whether the proxy class was loaded during getProxyFactory() or later when newInstance(Callback[]) is invoked during the getProxy(). With regard to the cglib case versus javassist. I need to investigate the difference more. I tested a new variation of the patch that should have better performance. See attached CGLIBLazyInitializer.patch3 + CGLIBProxyFactory.patch3
yes - looks like expected. Last remaining questions are:
1) would the javaassist version benefit from the same performance fix ? 2) will the earlier creation of the factory affect which classloader it will use to create new instances with ? 1. No, javassist is already just as optimized. the javassist class.newInstance() works correctly, so we do not need to create the instance of the class that will be used to create the proxy instance (the object returned by class.newInstance works fine).
2. I'm still trying to answer this question. Clearly, the classloader early in both cases to load the generated proxy class. I need to compare the old code versus the new code. The old code was something like: Class proxyFactoryClass=...; // completed early in getProxyFactory() // we need to understand the classloader use // in the following call. Object proxy = proxyFactoryClass.newInstance(); // completed late in getProxy() The new code is: Class proxyFactoryClass=...; // completed early in getProxyFactory() Object proxyFactory = proxyFactoryClass.newInstance(); // completed early in getProxyFactory // we need to understand the classloader use // in the following call. Object proxy = proxyFactory.newInstance(Callback[]); // completed late in getProxy() I'm currently trying to understand how to follow the classloader use in the old and new cases illustrated above. Clearly, the proxyFactory.newInstance(Callback[]) is hitting code that we generated in the proxy class. What is not clear is where the code for proxyFactoryClass.newInstance() lives. Is it the standard Class.newInstance() implementation code or do we somehow overide newInstance() in our generated proxy class. By the way, I talked briefly to Sun support about this issue and they are not aware of the jvm bug mentioned above. I would like to get more information about the failure case that could be relayed to Sun. If this issue is caused by a Sun bug, we should give them a standalone test case that reproduces the issue. We can lobby them to fix it then. If this has already be been done by someone, please let me know. I think that the same issue was reported on cglib forum http://sourceforge.net/forum/forum.php?thread_id=1441197&forum_id=190809 This patch will slow down startup, proxy class initialization is heavy weight and unitialized class is used as factory for this reason. I sugest to replace MethodInterceptor callback with InvocationHandler. This callback is very similar to java.lang.reflect.InvocationHandler, it needs less resources to generate and it will generate simplified proxies, without code generation in proxy <clinit> method.
ah - now i remember. we postpone the factory creation to when it is actually needed to mainly avoid being bogged down by static blocks in classes and generally to avoid doing too much in startup.
Juozas,
I don't completely follow this as I need to proxy the user entity class. I tried switching to invoking Proxy.newProxyInstance(Thread.currentThread().getContextClassLoader(), proxyInterfaces, instance); but I don't have an interface representing the entity class to pass in on the newProxyInstance method call. I end up getting a class cast exception in the client code when it tries to use the value returned by Proxy.newProxyInstance. Should I post my code changes as a patch for you to examine? Or is switching to InvocationHandler not going to work since I don't have an interface to pass in? Thanks for your advice! Max,
First to avoid confusion (on my part), we are talking about two possible paths to fixing this issue. 1. Making the current approach work correctly on Linux via CGLIBLazyInitializer.patch3. 2. Switching to InvocationHandler as suggested by Juozas. Regarding CGLIBLazyInitializer.patch3 and static blocks in classes, I assume they were being invoked in the old code during getProxyFactory time (at class load time). In the new implementation, we should still execute the static blocks in getProxyFactory (at class load time). Unless cglib is doing something that I woujldn't expect with regard to static blocks. Some of our choices for moving forward are: A. Use the CGLIBLazyInitializer.patch3 or some form of it to make the current approach function correctly. B. Switch to using InvocationHandler (I'm not sure if that will work for us since this can only proxy interfaces but not the user entity class). I'm hoping that I made a mistake here and that there is a different way to handle this. :) C. I don't want to do this but we could defer the "Object proxyFactory = proxyFactoryClass.newInstance()" to getProxy time and make this staticly synchronized on the CGLIBLazyInitializer class (defering initialization as late as possible at the cost of lower concurrency.) As a server guy, I'm against this approach, but it would be okay for non-server environments. Thoughts? startup time is highly affected in large projects if we have to call that static block even though we are not using it. That should be avoided!
The above is not dependent on fixing the cglib issue on Linux is it? It looks like the static initializer is invoked when we create the first instance of the class in getProxyFactory (in my patch#3). The following is a stack dump from the class static initializer of ManySessions test:
at java.lang.Thread.dumpStack(Thread.java:1158) at SystemParameter.<clinit>(SystemParameter.java:14) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:494) at java.lang.Class.newInstance0(Class.java:350) at java.lang.Class.newInstance(Class.java:303) at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxyFactory(CGLIBLazyInitializer.java:128) at org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstantiate(CGLIBProxyFactory.java:43) at org.hibernate.tuple.PojoEntityTuplizer.buildProxyFactory(PojoEntityTuplizer.java:159) at org.hibernate.tuple.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:131) at org.hibernate.tuple.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:50) at org.hibernate.tuple.TuplizerLookup.create(TuplizerLookup.java:64) at org.hibernate.tuple.EntityMetamodel.<init>(EntityMetamodel.java:256) at org.hibernate.persister.entity.AbstractEntityPersister.<init>(AbstractEntityPersister.java:424) at org.hibernate.persister.entity.SingleTableEntityPersister.<init>(SingleTableEntityPersister.java:109) at org.hibernate.persister.PersisterFactory.createClassPersister(PersisterFactory.java:55) at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1218) at ManySessionsTest.<clinit>(ManySessionsTest.java:26) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Bug
Resolved
Blocker
java.lang.NoSuchMethodError: <persistent class>.getHibernateLazyInitializer()
