Quote:
Originally Posted by andw
Haven't been able to run the threaded versions, seem to end in a race / lock condition. The log window shows one worker for a while then the other with an increasing pause between them swapping before the gui becomes unresponsive. Memory usage goes from ~120MB to ~350MB and CPU usage drops to zero.
Currently using 1.17 although I'll have to change the 1000 post limit soon.
|
tricky...
The GUI should not become unresponsive. I found one probable cause for this. The updating of the log view is not thread safe. This I can fix.
I currently have problems downloading from Imagebam. I get Connect timeouts. Relevant lines fro the log:
Code:
vef-imagerescue.log.0:2017-06-19 23:46:54 INFO: image-page: [ForkJoinPool-1-worker-1] http://www.imagebam.com/image/43330256412819
vef-imagerescue.log.0:2017-06-19 23:49:02 SEVERE: Error downloading 'http://www.imagebam.com/image/43330256412819'
org.apache.http.conn.HttpHostConnectException: Connect to 5.imagebam.com:80 [5.imagebam.com/199.58.85.103] failed: Connection timed out (Connection timed out)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
Worker 1 tries to download 43330256412819 at 23:46:54. A connect is initiated but three minutes later at 23:49:02 the connect is aborted because of a timeout. There are no log entries from worker 1 in this period. This could match your description.
My plan is
- Fix the probable cause of GUI freezing
- Experiment with timeouts. There is seemingly a default of 3 minutes which is definitely too long. I am thinking more of 10 seconds.
- Make the parsing for posts more resilient (deleted posts problem).
I had no problems with multi threading until I got this Connect Exceptions. But I must admit that I usually do not use the GUI.
addendum: after setting the timeout to 10 seconds both workers log connection timeouts every 10 seconds. I will try again in a few hours.