Selenium Twitter Bot - Automation Demo with Python

Today we are going to learn how to use Selenium and Python to automate Twitter using Firefox.

WARNING - If you are automating your interactions with Twitter it can be easy to break the rules and have your account suspended. Be careful what you do.

If you don’t already have Selenium setup on your system you will probably want to take a look at our other guide first: Selenium Setup and Install Webdriver with Python.

Using the XPath

When automating things with Selenium you will need to be able to refer to specific elements on a webpage ( ex: a text field or submit button ). One of the easiest and most covient ways of doing this is specicfy what you want to interact with based on the xpath of that element.

Here is an example of what the xpath currently looks like for the password field on the Twitter login page:


/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[2]/label/div/div[2]/div/input

To find the xpath of an element using Firefox follow these steps:

Right click on the element and select “inspect”. This will open a debug panel with the element highlighted in the Inspector tap.
Right click on the highlighed bit of code for the element.
Goto “Copy” on the menu.
Select “XPath” on the submenu. This will copy the xpath into your clipboard.
Paste the xpath into your code where needed.

WARNING - The xpath for an element on a given webpage will change when that site is updated. You will likely find yourself having to update your scripts with new xpaths from time to time.

Key Components

If you just want to see a full working example keep scrolling down past this section.

Here we are mostly going to cover the key, Selenium specific components of our script.

This isn’t a Selenium function but it should be pointed out. We frequently make use of the sleep function for timing. This is important for a couple of reasons. First of all we need to make sure that the page is loaded before we interact with it. Seond, if we interact too quickly it will cause our software to look like a bot. You may want your script to hide this and pretend to be a real person.


time.sleep(2)

This loads the Firefox driver:


driver = webdriver.Firefox()

This loads the URL specified. In this case it is Twitter.


driver.get("http://twitter.com/login")

This will wait for the title to contain a specific string. If you know what title to expect this is a good indication that a page has loaded. We also specify a timemout value of 10 seconds.


WebDriverWait(driver, 10).until(EC.title_contains("Home"))

Here we lookup an element on the page based on the XPath of that element.


inputElement2 = driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[2]/label/div/div[2]/div/input")

Here we send send keystrokes to an element on the page ( an input field ). The string passed contains the keys to send.


inputElement2.send_keys(password)

We can submit a form like this:

    
inputElement1.submit()

We can use this to scroll the page down. This is important because some sites will not load elements until you actually scroll down to them. This seems to be the case with Twitter.


driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

We can use this to quit. This will kill the browser.


driver.quit();

Full Example: Selenium / Python / Twitter

This is a full example with all of the parts put together to do something semi-useful. Here is what it does:

Read username and password from an external file ( not hardcoded in the script ).
Login to twitter
Checks the profile and checks the number of followers as well as the number of other people being followed.
Does a search for a topic and record the usernames for the top posts.

Note that we make frequent use of the sleep() function. Timing is important for a couple of reasons.

The page need to load before you interact with it.
If you interact too quickly that can tip off the site that you are running bot software.

test2.py

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0
import time
import re
from os.path import expanduser


driver = webdriver.Firefox()


home_dir = expanduser("~")

user = ""
password = ""


def read_creds():
    global user
    global password
    with open(home_dir + "/.planetary_vegetation/creds.dat") as f1:
        data = f1.read()
        x = data.split("\n")
        user = x[0]
        password = x[1]

def login():
    driver.get("http://twitter.com/login")

    print( driver.title )
    time.sleep(5)
    inputElement1 = driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[1]/label/div/div[2]/div/input")
    inputElement2 = driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[2]/label/div/div[2]/div/input")
    time.sleep(2)
    inputElement1.send_keys(user)
    time.sleep(2)
    inputElement2.send_keys(password)
    time.sleep(2)
    inputElement1.submit()

    try:
        WebDriverWait(driver, 10).until(EC.title_contains("Home"))
        print( driver.title )
    finally:
        pass

def stats():
    following = "0"
    followers = "0"
    driver.get("https://twitter.com/Stegosa99042135")
    time.sleep(5)
    #followers = driver.find_element_by_link_text("Followers")
    data1 = driver.find_element_by_xpath("/html/body/div")
    m1 = re.search(data1.text, "\n(\d+) Following\n(\d)+Followers\n")
    if m1:
        following = m1.group(1)
        followers = m1.group(2)
    print("Following: " + following)
    print("Followers: " + followers)


def search(search_term):
    links = []

    driver.get("https://twitter.com/search?q=" + search_term + "&src=typed_query")
    time.sleep(5)


    users_who_posted = []
    users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[1]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text)
    users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[2]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)
    users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[7]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text)
    for i in range(3,20):
        try:
            users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[" + str(i) + "]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text)
        except: 
            pass

    print(users_who_posted)



read_creds()
login()
time.sleep(2)
stats()
time.sleep(2)
search("kittens")   # returns list of user profiles that posted a search



time.sleep(1000)
driver.quit();

Video Guide

This is the video guide showing how to setup and use what we have covered on this page.