Sending e-mail after scrape in scrapy


Sending e-mail after scrape in scrapy code
class Examplepipeline(object):

def __init__(self):
dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
dispatcher.connect(self.spider_closed, signal=signals.spider_closed)

def spider_opened(self, spider):
log.msg(“opened spider %s at time %s” % (,‘%H-%M-%S’)))

def process_item(self, item, spider):
log.msg(“Processsing item ” + item[‘title’], level=log.DEBUG)

def spider_closed(self, spider):
log.msg(“closed spider %s at %s” % (,‘%H-%M-%S’)))

In the above spider code , it will display the starting time and ending time of the spider, but now after the completion of the spider, i want to receive a mail that “Scraping has been completed” from scrapy. Is it possible to do this. If possible can we write that code in spider_closed method, can anyone please share some example code on how to do this.


Solution 1:

Have you looked into documentation:

Basic usage from documentation

from scrapy.mail import MailSender

mailer = MailSender()
mailer.send(to=[""], subject="Some subject", body="Some body", cc=[""])

Also you could implement something custom on your own. For example if you want to use gmail:

def send_mail(self, message, title):
    print "Sending mail..........."
    import smtplib
    from email.MIMEMultipart import MIMEMultipart
    from email.MIMEText import MIMEText
    gmailUser = ''
    gmailPassword = 'password'
    recipient = 'mail_to_send_to'

    msg = MIMEMultipart()
    msg['From'] = gmailUser
    msg['To'] = recipient
    msg['Subject'] = title

    mailServer = smtplib.SMTP('', 587)
    mailServer.login(gmailUser, gmailPassword)
    mailServer.sendmail(gmailUser, recipient, msg.as_string())
    print "Mail sent"

and just call it like:

send_mail("some message", "Scraper Report")

Solution 2:

My apologies for self-promotion, but I recently created yagmail: a package that strives to make it easy to send gmail messages (text, html, image etc).

This is the code you’d need to connect:

import yagmail
yag = yagmail.SMTP('', 'password')

Then you use this to send emails:

yag.send('mail_to_send_to', 'Scraper Report', 'some message')

What is nice is that you do not have to keep settings around as text, but you can rely on the OS’ keyring for a real safe and comfortable feeling.

It could even be a one liner (that closes automatically):

SMTP('mail_you_send_from').send('mail_to_send_to', 'Scraper Report', 'some message')