Bug#933857: solr-jetty: Jetty lacks necessary write permissions to /var/lib/solr/data/index/

beirer debian at lowmem.net
Sun Aug 4 14:31:52 BST 2019 

Package: solr-jetty
Version: 3.6.2+dfsg-20
Severity: important

Hi Maintainers,

After upgrdaing from Stretch to Buster I experienced the following
problem:

Jetty lacks write permissions to /var/lib/solr/data/index, which results
in exceptions for some SOLR operations.

To reproduce:

/usr/bin/curl -s http://localhost:8080/solr/update?optimize=true

journalctl -u jetty9.service: 

Aug 04 14:21:42 srv1 jetty9[700]: Aug 04, 2019 2:21:42 PM org.apache.solr.common.SolrException log
Aug 04 14:21:42 srv1 jetty9[700]: SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/lib/solr/data/index/write.lock: java.io.FileNotFoundException: /var/lib/solr/da
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.lucene.store.Lock.obtain(Lock.java:84)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:84)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:101)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:171)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:404)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:52)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.Server.handle(Server.java:502)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
Aug 04 14:21:42 srv1 jetty9[700]:         at java.base/java.lang.Thread.run(Thread.java:834)
Aug 04 14:21:42 srv1 jetty9[700]: Caused by: java.io.FileNotFoundException: /var/lib/solr/data/index/write.lock (Read-only file system)
Aug 04 14:21:42 srv1 jetty9[700]:         at java.base/java.io.RandomAccessFile.open0(Native Method)
Aug 04 14:21:42 srv1 jetty9[700]:         at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
Aug 04 14:21:42 srv1 jetty9[700]:         at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
Aug 04 14:21:42 srv1 jetty9[700]:         at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:203)
Aug 04 14:21:42 srv1 jetty9[700]:         at org.apache.lucene.store.Lock.obtain(Lock.java:95)
Aug 04 14:21:42 srv1 jetty9[700]:         ... 40 more




This seems to be very similar to Bug #919638 "solr-tomcat: Permission problems after update to tomcat9" (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919638).
The fix is also similar:

Copy /lib/systemd/system/jetty9.service to /etc/systemd/system and modify
it by adding


ReadWritePaths=/var/lib/solr/data

to the # Security paragraph.

Restart jetty:

systemctl restart jetty9.service




BTW, the modification in /etc/solr/solr-jetty.xml attached below is due
to bug #933854 (reported by me) and is probaly unrelated to above
behaviour, I attached it for completeness.

HTH, best regards,

stephan


-- System Information:
Debian Release: 10.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-5-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages solr-jetty depends on:
ii  default-jdk [java5-sdk]     2:1.11-71
ii  jetty9                      9.4.15-1
ii  libjetty9-extra-java        9.4.15-1
ii  openjdk-11-jdk [java5-sdk]  11.0.3+7-5
ii  solr-common                 3.6.2+dfsg-20

solr-jetty recommends no packages.

solr-jetty suggests no packages.

-- Configuration Files:
/etc/solr/solr-jetty.xml changed:
<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">
<!-- Context configuration file for the Solr web application in Jetty -->
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="contextPath">/solr</Set>
  <Set name="war">/usr/share/solr/web</Set>
  <!-- Set the solr.solr.home system property -->
  <Call name="setProperty" class="java.lang.System">
    <Arg type="String">solr.solr.home</Arg>
    <Arg type="String">/usr/share/solr</Arg>
  </Call>
  <!-- Enable symlinks -->
  <!-- Disabled um Problemen beim SOLR-Start aus dem Weg zu gehen 
  <Call name="addAliasCheck">
    <Arg>
      <New class="org.eclipse.jetty.server.handler.ContextHandler$ApproveSameSuffixAliases"/>
    </Arg>
  </Call>
  -->
</Configure>


-- no debconf information

More information about the pkg-java-maintainers mailing list