Low Orbit Flux Logo 2 F

Selenium Twitter Bot - Automation Demo with Python

Today we are going to learn how to use Selenium and Python to automate Twitter using Firefox.

WARNING - If you are automating your interactions with Twitter it can be easy to break the rules and have your account suspended. Be careful what you do.

If you don’t already have Selenium setup on your system you will probably want to take a look at our other guide first: Selenium Setup and Install Webdriver with Python.

Using the XPath

When automating things with Selenium you will need to be able to refer to specific elements on a webpage ( ex: a text field or submit button ). One of the easiest and most covient ways of doing this is specicfy what you want to interact with based on the xpath of that element.

Here is an example of what the xpath currently looks like for the password field on the Twitter login page:


/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[2]/label/div/div[2]/div/input

To find the xpath of an element using Firefox follow these steps:

WARNING - The xpath for an element on a given webpage will change when that site is updated. You will likely find yourself having to update your scripts with new xpaths from time to time.

Key Components

If you just want to see a full working example keep scrolling down past this section.

Here we are mostly going to cover the key, Selenium specific components of our script.

This isn’t a Selenium function but it should be pointed out. We frequently make use of the sleep function for timing. This is important for a couple of reasons. First of all we need to make sure that the page is loaded before we interact with it. Seond, if we interact too quickly it will cause our software to look like a bot. You may want your script to hide this and pretend to be a real person.


time.sleep(2)

This loads the Firefox driver:


driver = webdriver.Firefox()

This loads the URL specified. In this case it is Twitter.


driver.get("http://twitter.com/login")

This will wait for the title to contain a specific string. If you know what title to expect this is a good indication that a page has loaded. We also specify a timemout value of 10 seconds.


WebDriverWait(driver, 10).until(EC.title_contains("Home"))

Here we lookup an element on the page based on the XPath of that element.


inputElement2 = driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[2]/label/div/div[2]/div/input")

Here we send send keystrokes to an element on the page ( an input field ). The string passed contains the keys to send.


inputElement2.send_keys(password)

We can submit a form like this:

    
inputElement1.submit()

We can use this to scroll the page down. This is important because some sites will not load elements until you actually scroll down to them. This seems to be the case with Twitter.


driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

We can use this to quit. This will kill the browser.


driver.quit();

Full Example: Selenium / Python / Twitter

This is a full example with all of the parts put together to do something semi-useful. Here is what it does:

Note that we make frequent use of the sleep() function. Timing is important for a couple of reasons.

test2.py
from selenium import webdriver from selenium.common.exceptions import TimeoutException from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0 from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0 import time import re from os.path import expanduser driver = webdriver.Firefox() home_dir = expanduser("~") user = "" password = "" def read_creds(): global user global password with open(home_dir + "/.planetary_vegetation/creds.dat") as f1: data = f1.read() x = data.split("\n") user = x[0] password = x[1] def login(): driver.get("http://twitter.com/login") print( driver.title ) time.sleep(5) inputElement1 = driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[1]/label/div/div[2]/div/input") inputElement2 = driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div[2]/form/div/div[2]/label/div/div[2]/div/input") time.sleep(2) inputElement1.send_keys(user) time.sleep(2) inputElement2.send_keys(password) time.sleep(2) inputElement1.submit() try: WebDriverWait(driver, 10).until(EC.title_contains("Home")) print( driver.title ) finally: pass def stats(): following = "0" followers = "0" driver.get("https://twitter.com/Stegosa99042135") time.sleep(5) #followers = driver.find_element_by_link_text("Followers") data1 = driver.find_element_by_xpath("/html/body/div") m1 = re.search(data1.text, "\n(\d+) Following\n(\d)+Followers\n") if m1: following = m1.group(1) followers = m1.group(2) print("Following: " + following) print("Followers: " + followers) def search(search_term): links = [] driver.get("https://twitter.com/search?q=" + search_term + "&src=typed_query") time.sleep(5) users_who_posted = [] users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[1]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text) users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[2]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text) driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(2) users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[7]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text) for i in range(3,20): try: users_who_posted.append( driver.find_element_by_xpath("/html/body/div/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div[" + str(i) + "]/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div[1]/div[1]/div[1]/a/div/div[2]/div/span").text) except: pass print(users_who_posted) read_creds() login() time.sleep(2) stats() time.sleep(2) search("kittens") # returns list of user profiles that posted a search time.sleep(1000) driver.quit();

Video Guide

This is the video guide showing how to setup and use what we have covered on this page.