robots.txt depends on ALLOW_ROBOTS env variable

robots.txt depends on runtime ALLOW_ROBOTS env variable

send robots.txt in response

removed unnecessary file and module
4 jobs for #6_robots_txt in 3 minutes and 19 seconds (queued for 2 minutes and 13 seconds)
Status Job ID Name Coverage
  Package
passed #3465
package the branch

00:02:42

 
  Dockerize
passed #3466
dockerize the branch

00:00:13

 
  Deploy
passed #3467
manual
deploy for review

00:00:11

 
  Undeploy
passed #3468
manual
remove review instance

00:00:11