python - Logging to specific error log file in scrapy -

i running log of scrapy doing this:

from scrapy import log class myspider(basespider):   name = "myspider"      def __init__(self, name=none, **kwargs):         log_file = "logs/spider.log"         log.log.defaultobserver = log.log.defaultobserver()         log.log.defaultobserver.start()         log.started = false         log.start(log_file, loglevel=log.info)         super(myspider, self).__init__(name, **kwargs)      def parse(self,response):         ....         raise exception("something went wrong!")         log.msg('something went wrong!', log.error)          # somehow write separate error log here. 

then run spider this:

scrapy crawl myspider 

this store log.info data log.error spider.log.

if error occurs, store details in separate log file called spider_errors.log. make easier search errors occurred rather trying scan through entire spider.log file (which huge).

is there way this?

edit:

trying pythonloggingobserver:

def __init__(self, name=none, **kwargs):         log_file = 'logs/spider.log'         err_file = 'logs/spider_error.log'          observer = log.log.pythonloggingobserver()         observer.start()          log.started = false              log.start(log_file, loglevel=log.info)         log.start(err_file, loglevel=log.error) 

but error: no handlers found logger "twisted"

just let logging job. try use pythonloggingobserver instead of defaultobserver:

• configure 2 loggers (one info , 1 error messages) directly in python, or via fileconfig, or via dictconfig (see docs)

• start in spider's __init__:

  def __init__(self, name=none, **kwargs):     # todo: configure logging: e.g. logging.config.fileconfig("logging.conf")     observer = log.pythonloggingobserver()     observer.start() 

let me know if need configuring loggers.

edit:

another option start 2 file log observers in __init__.py:

from scrapy.log import scrapyfilelogobserver scrapy import log   class myspider(basespider):     name = "myspider"        def __init__(self, name=none, **kwargs):         scrapyfilelogobserver(open("spider.log", 'w'), level=logging.info).start()         scrapyfilelogobserver(open("spider_error.log", 'w'), level=logging.error).start()          super(myspider, self).__init__(name, **kwargs)      ...