QSirch reindexing in loop

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

QSirch reindexing in loop

Post by nikonvulcano » Wed Dec 26, 2018 5:22 pm

Hi
since first installation (just presented) QSirch had always issues, for me.
I have also reinitialize the NAS and reinstall QSirch, but

1-if reboot the NAS or update the firmware, the indexing restart from begin as the first time (taking 3-5 days to terminate). This each time !!
2-the CPU load during the indexing is about 50% only for QSirch process, than low to 20-30% when it terminate.

My NAS is a TVS-463 with 16GB of RAM.

I opened also a ticket but the help desk told me that is difficult to replicate my issue, and since months I'm still rest in the same situation.

Can anybody help me?
thanks
bye
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Sun Jan 27, 2019 3:13 am

I assume that ES and/or Tika run out of memory. For me the ES max java heap memory is set to 256 MB which may be sufficient for 50.000-100.000 documents, depending on the type and metadata.
You could try to edit these configuration files and change 256 to 1024 and restart then Qsirch.
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - MAX_ES_HEAP=256
'ps -ef|grep elastic' should then list the ES process with '-Xmx1024M'. If you modified also Tika_Mem_Max it could fix also Tika.

'grep "java.lang.OutOfMemoryError" /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/tika_server.log' should show Tika problems.

Edited: -ES_MAX_HEAP +MAX_ES_HEAP
Last edited by iam@nas on Sat Feb 02, 2019 7:00 pm, edited 1 time in total.

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Tue Jan 29, 2019 4:48 am

Hi iam@nas,
thank you for your reply.

I've checked the Tika process and I see that some times it uses a lot of CPU.
My TVS-463 has 16 GB of RAM.

I don't think that my problem is an out of memory, but I'm not an expert.

Please, look at the screenshot in attachment...

What can you tell me?

Thanks
Bye
You do not have the required permissions to view the files attached to this post.
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Wed Jan 30, 2019 3:06 am

Open a shell and locate the 'tika_server.log' file. It should be in the '/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/' folder.

Code: Select all

cd /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/
grep "OutOfMem" tika_server.log
If it returns 'Exception in thread "Thread-1234" java.lang.OutOfMemoryError: Java heap space' there was a memory problem. There are some time stamps in the log which are not related to this error. So one can try to guess when the problem occured. It may be a good idea to stop Qsrich in the web UI, rename the log file

Code: Select all

mv tika_server.log tika_server.log.old
and start it again to get a clean log file.

I have no extractors running at the moment. So many running extractors allocating all CPU resources seems to be bad.

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Wed Jan 30, 2019 4:59 am

Hi iam@nas,
thanks for you reply.

I tried to do as you suggest, but I receive

Binary file tika_server.log matches

Please look at the attachment.

However, before to rename the log, I wait in this days the check from RD of helpdesk.
Moreover to complete the indexing it needs some days.

Why the grep don't have worked for me?

thanks
bye
You do not have the required permissions to view the files attached to this post.
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Wed Jan 30, 2019 10:23 pm

It seems that your log file contains also some binary data. Or your grep / busybox has some issues with the file.
In any case the grep command matched. Your log file contains "OutOfMem" so there were some Tika memory issues some time ago.

I installed Entware-ng and then most of the standard unix tools (procps*, coreutils*). QTS also contains most of these tools within busybox and thus with a very limited functionality.

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Thu Jan 31, 2019 5:05 am

Hi iam@nas,
thank you again.
After intervention of helpdesk, I will delete the log.

Me too, I've installed Entware to add some tool, but I didn't know of procps*, coreutils* ...

thank you for the precious infos !!
bye
TVS-463 - 16GB RAM - QTS 4.3.6

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Sat Feb 02, 2019 6:38 pm

Hi iam@nas
I tried your suggestion. I done all

iam@nas wrote:
Sun Jan 27, 2019 3:13 am
I assume that ES and/or Tika run out of memory. For me the ES max java heap memory is set to 256 MB which may be sufficient for 50.000-100.000 documents, depending on the type and metadata.
You could try to edit these configuration files and change 256 to 1024 and restart then Qsirch.
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - ES_MAX_HEAP=256
'ps -ef|grep elastic' should then list the ES process with '-Xmx1024M'. If you modified also Tika_Mem_Max it could fix also Tika.

'grep "java.lang.OutOfMemoryError" /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/tika_server.log' should show Tika problems.

modified to 1024 both
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - ES_MAX_HEAP=256

restarted Qsirch

but when i do 'ps -ef|grep elastic' I don't receive ES process with '-Xmx1024M' but

Code: Select all

9437 admin      2440 S   grep elastic
22202 admin    379020 S < /share/CACHEDEV1_DATA/.qpkg/JRE/jre/bin/java -Xms256M -Xmx512M -Xss256k -Djava.awt.headless=true -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch -cp :/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/elasticsearch-1.4.2.jar:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/*:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/sigar/* -Des.logger.level=INFO -Des.path.logs=/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/es_logs -Des.http.port=5028 -Des.network.host=127.0.0.1 -Des.transport.tcp.port=5029 org.elasticsearch.bootstrap.Elasticsearch
Note that my Qsirch, now, is v.4 version.

Please can you help me again?
thanks
bye
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Sat Feb 02, 2019 6:58 pm

Looks like I made a typo. Did you modify "MAX_ES_HEAP=256" to "MAX_ES_HEAP=1024" in qusion.sh?
Or did you add ES_MAX_HEAP=256 ? If so please remove it as it has no effect.

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Sat Feb 02, 2019 7:15 pm

hi,
I think I did everything correctly, please also look at the attached screenshot
thanks
You do not have the required permissions to view the files attached to this post.
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Sat Feb 02, 2019 8:22 pm

Was the file Qsirch/supervisord.conf re-created during startup?

Code: Select all

$ ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
should show the current / last day as date.

$ grep "m:ela" -A 5 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
should output:
[program:elasticsearch]
command=[...]
environment=JAVA_HOME=/share/CACHEDEV1_DATA/.qpkg/JRE/jre,ES_MIN_MEM=512M,ES_MAX_MEM=1024M
directory=[...]
[...]


To get more details please add 3 lines at the beginning pf of qusion.sh so it starts like this:

Code: Select all

#!/bin/sh
set -x
exec >/tmp/qsirch.log
exec 2>&1

# Get java path ... all other code remains as is
Restarting Qsirch will create the file /tmp/qsirch.log
"grep ES_ /tmp/qsirch.log" should output

Code: Select all

+ MAX_ES_HEAP=1024
+ SYS_ALLOW_MAX_ES_HEAP=2048
...
+ /sbin/setcfg Global ES_Mem_Max 1024 -f /etc/config/qusion.conf
+ /sbin/setcfg Global Sys_Allow_ES_Mem_Max 2048 -f /etc/config/qusion.conf
...
+ ES_MIN_MEM=512M
+ ES_MAX_MEM=1024M
...
+ /bin/sed -i 's#((ES_MIN_MEM))#512M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MAX_MEM))#1024M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
...
WARNING 2019-02-02 12:59:58,882 - es_sync        : should_enlarge_heap_memory 97   - CURRENT_ES_MEM_MAX = 1024, SYS_ALLOW_ES_MEM_MAX = 2048

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Sat Feb 02, 2019 8:37 pm

Hi
something don't worked because I don't received your output after grep of supervisors.conf, but this:

ES_MIN_MEM=256M,ES_MAX_MEM=512M

the half values

Code: Select all

[program:elasticsearch]
command=/share/CACHEDEV1_DATA/.qpkg/Qsirch/bin/python -u -c 'import os; os.nice(-5); cmd = "/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/bin/elasticsearch.nas  -D es.logger.level=INFO -D es.path.logs=/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/es_logs -D es.http.port=5028 -D es.network.host=127.0.0.1 -D es.transport.tcp.port=5029".split(); os.execl("/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/bin/elasticsearch.nas", *cmd)'
environment=JAVA_HOME=/share/CACHEDEV1_DATA/.qpkg/JRE/jre,ES_MIN_MEM=256M,ES_MAX_MEM=512M
directory=/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/bin
autostart=false
exitcodes=0
iam@nas wrote:
Sat Feb 02, 2019 8:22 pm
Was the file Qsirch/supervisord.conf re-created during startup?

Code: Select all

$ ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
should show the current / last day as date.

$ grep "m:ela" -A 5 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
should output:
[program:elasticsearch]
command=[...]
environment=JAVA_HOME=/share/CACHEDEV1_DATA/.qpkg/JRE/jre,ES_MIN_MEM=512M,ES_MAX_MEM=1024M
directory=[...]
thanks
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Sat Feb 02, 2019 9:11 pm

Code: Select all

ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
-rw-r--r-- 1 admin administrators 11456 2019-02-02 11:59 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
Does it also show 2019-02-xx for you? Otherwise please stop and start Qsirch in the App Center.

nikonvulcano
Getting the hang of things
Posts: 65
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano » Sat Feb 02, 2019 9:20 pm

Sorry, I don't sure I understood correctly.

The hour and date are correct. I've already stop and restart in App Center, many times.

Code: Select all

ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
-rw-r--r-- 1 admin administrators 11455 2019-02-02 12:21 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
What do you suggest me?

Thanks
TVS-463 - 16GB RAM - QTS 4.3.6

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas » Sat Feb 02, 2019 9:23 pm

In viewtopic.php?f=361&t=145738#p704700 I posted the 'set -x' settings for qusion.sh to create /tmp/qsirch.log after startup. 'set -x' runs the script in verbose mode and in the created output one should see what's going on. It would be great if you could post it.
Last edited by iam@nas on Sat Feb 02, 2019 9:34 pm, edited 1 time in total.

Post Reply

Return to “Qsirch & Qfiling”