[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] Only block the bots on jenkins.d.n and tests.r-b.o, not the other websites

[Mattia Rizzolo]

Mattia Rizzolo pushed to branch master at Debian QA / jenkins.debian.net


Commits:
e09cda74 by Mattia Rizzolo at 2019-12-21T14:33:31Z
Only block the bots on jenkins.d.n and tests.r-b.o, not the other websites

This should let the crawlers crawl r-b.o, diffoscope.org, etc, which we
really want to be indexed everywhere.

Signed-off-by: Mattia Rizzolo <mattia at debian.org>

- - - - -


1 changed file:

- hosts/jenkins/etc/apache2/sites-available/jenkins.debian.net.conf


Changes:

=====================================
hosts/jenkins/etc/apache2/sites-available/jenkins.debian.net.conf
=====================================
@@ -15,10 +15,7 @@
 	Header always add Strict-Transport-Security "max-age=15552000"
 </Macro>
 
-<Macro common-directives $name>
-	ServerName $name
-	ServerAdmin holger at layer-acht.org
-
+<Macro block-bots>
 	#block bad bots with a 403
 	SetEnvIf User-Agent "AhrefsBot" bad_bot
 	SetEnvIf User-Agent "Baiduspider" bad_bot
@@ -39,7 +36,11 @@
 	SetEnvIf User-Agent "Twitterbot" bad_bot
 	SetEnvIf User-Agent "yacybot" bad_bot
 	SetEnvIf User-Agent "ZoomBot" bad_bot
-	#end block bad bots
+</Macro>
+
+<Macro common-directives $name>
+	ServerName $name
+	ServerAdmin holger at layer-acht.org
 
 	<Directory />
 		Options FollowSymLinks
@@ -129,6 +130,7 @@ Use https-redirect www.diffoscope.org
 <VirtualHost *:443>
 	Use common-directives jenkins.debian.net
 	Use common-directives-ssl jenkins.debian.net
+	Use block-bots
 
 	DocumentRoot /var/www
 	AddDefaultCharset utf-8
@@ -189,6 +191,7 @@ Use https-redirect www.diffoscope.org
 <VirtualHost *:443>
 	Use common-directives tests.reproducible-builds.org
 	Use common-directives-ssl tests.reproducible-builds.org
+	Use block-bots
 
 	DocumentRoot /var/lib/jenkins/userContent/reproducible
 	AddDefaultCharset utf-8



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/commit/e09cda7411fd293090e9d8fd88c25ffb361248c2

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/commit/e09cda7411fd293090e9d8fd88c25ffb361248c2
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20191221/37f6ef2b/attachment-0001.html>