Python Tutorial 2.7 - Part 1

Bitwise Operators

These are great if you want to manipulate things on the bit level. This is useful for many different things such as encoding and decoding raw network packets.

<<

>>

2 + 2
50 - 5*6
(50 - 5.0*6) / 4
23 / 4      # results in a float
23 / 4.0    # results in a float
23 // 4.0   # explicit floor division - discards remainder
23 % 4      # modulus - remainder after division
7 ** 2      # 7 squared
2 ** 15     # 2 to the power of 15

a = 20
b = 5 * 9
c = a * b + 45

type(a)

a = 5
b = 5.0
y = 420 ** 99
x = 3 + 4j
s = "test"
print(type(a))  # int
print(type(b))  # float
print(type(y))  # long
print(type(x))  # complex
print(type(s))  # string


f = (1,2)
g = [1,2]
h = {1:"a", 2:"b"}
i = {1, 2, 3}
j = False
print(type(f))  # tuple
print(type(g))  # list
print(type(h))  # dict
print(type(i))  # set
print(type(j))  # bool

a = 4
b = 4.5
c = a + b
print(c)
print(type(a))   # int
print(type(b))   # float
print(type(c))   # float

Strings

"example text"       #  double quotes work
'example text'       #  single quotes work

'doesn\'t'           # escape a single quote with a \'
"doesn't"            # don't need to escape 

"test \"a\" here"    # escape double quotes with a \
'test "a" here'      # don't need to escape 

'test \'a\' here'    # not needed, won't work as 
                     # an escape character

x = "line one\nline two"
y = 'line one\nline two'

Raw Strings

Using a raw string, escape characters will not be recognized. To create a raw string, just precede it with an 'r'.

'test \n test'        # normal string -  \n is a new line
r'test \n test'       # raw string - \n is litteral

Escape Characters for Strings

Escape Characters for Unicode Strings

Multiline Strings

Multiline string scan be constructed. You can use either triple double quotes or triple single quotes. Both work but you should use double quotes for your triple quoted strings. This will maintain consistency with the docstring convention: SEE HERE

test = """\
This is a test string.\ 
Blah blah blah
Blah blah blah
"""

test = '''\
This is a test string.\ 
Blah blah blah
Blah blah blah
'''

Note that this will not require '\n' for each new line. Those will just work. The first slash at the end of the first line is optional but will escape the first newline character. The slash at the end of the second line is treated literally and will not escape anything.

Python does not actually support multiline comments. Using a triple quoted string without storing the value can achieve the same result.

"""
My comment goes here.
This is really a string.
"""

Concat and Repeat

x = 5 * 'b'              # results in bbbbb

x = 'abc' + 'def'

x = 3 * 'a' + 'b'             # results in aaab

x = 'abc' 'def'           # results in abcdef
x = 'abc' 'def' + x       # also works 
x = 'abc' 'def'  x        # ERROR - need '+' for a var

x = ('Some example text '
          'and more text here')

a = 'abc'
b = 23
x = 'test' + a        # works - a is a string
x = 'test' + b             # ERROR - can't concat an integer
x = 'test' + str(b)       # works - convert b to string first

Note - if you just printing things for debugging purposes and don't need to save the string, you can use this shortcut:

b = 23
print 'test', b      # works, automatically adds a space
print 'one', 'two'   # also, automatically adds a space

String Indexing

s = 'Python'
s[0]         # 'P' - first char
s[5]         # 'n' - 6th char
s[-1]        # last character - 'n'
s[-2]      # second to last character 'o'

String Slicing

Strings can be sliced. The starting index is included, the ending index is excluded.

s[0:2]  # char 0 to 2 (excludes 2)  'Py'
s[2:5]  # char 2 to 5 (excludes 5)  'tho'
s[:i] + s[i:]      # entire string


s[:2]     #    'Py'
s[4:]     #    'on'
s[-2:]    #    'on'

• Slice length = first index - second index
• Length of s[1:3] is 2

s = 'Python'
s[42]              # out of bounds error
s[4:42]    # no error, ignores extra chars ( 'on' )
s[42:]       # no error, empty string

s[0] = 'J'    # error

• To change a string, create a new one.

len(s)

Unicode

u'Hello World !'
u'Hello\u0020World !'

ur'Hello\u0020World !'
ur'Hello\\u0020World !'

u"abc"          # prints out string
str(u"abc")    # can convert to string
u"äöü"          # prints out bytes in hex
str(u"äöü")    # can't convert to string

u"äöü".encode('utf-8')          

unicode('\xc3\xa4\xc3\xb6\xc3\xbc', 'utf-8')  

Lists

• These are also called arrays in many languages. In Python they are called lists.
• Lists are mutable. The elements can be updated.
• They are usually homogeneous but don't have to be.

a = [1, 4, 9, 16, 25, 1, 2, 3, 4, 5]     # define and initialize
a[0]           # return item at index 0
a[3] = 64      # assign a value
a.append(216)  # append a value to the end
a[-1]          # return item at last index
a[-2]          # return item at second to last index


a + [36, 49, 64, 81, 100]    # concatenate and return a new list

a[2:7]    # slicing returns a new list, shallow copy
a[:7]     # begining to index 7
a[3:]     # index 3 to end
a[:]      # this returns a slice of the entire list


a[2:5] = ['x', 'y', 'z']     # replace slice
a[2:5] = []                  # remove this slice

len(a)

a = [0, 1, 2, 3, 4]
del a[0]           # delete an element from a list
del a[5:75]        # delete a slice from a list
del a[:]           # clear a list
del a              # delete the entire variable

a = ['a', 'b', 'c']
n = [1, 2, 3]
x = [a, n]           # 2D - nested list
x[0]                 # returns a list
x[0][1]              # returns element form nested list

List Functions

Flow Control

x = raw_input("What is your name? ")
x = int(raw_input("How old are you? "))

if/elif/else:

if x == 5:
   print 'It matches!'
elif x < 5:
   print('It\'s less.')
elif x > 5:
   print('It\'s more.')
else:
   print 'something else...'

x = 'b'
if x in ('a', 'b', 'c', 'test'):
   print("match")

m = 'test'
d = 'abc'

# both of these has to be true
if m == 'test' and d == 'xyz':
   print("test 1")

# only one of these needs to be true
if m == 'test' or d == 'xyz':
   print("test 2")

# both of the first two OR the third 
if m == 'test' and m != 'abc' or d == 'xyz':
   print("test 3")

The while loop

A while loop just keeps looping as long as the condition is true.

c = 0
a = 1
b = 2

while c < 25:
   print c
   c = c + a + b

while 5 == 5:
    print "."

keep_going = True
while keep_going == True:
    print "."

while True:
    print "."

The for loop:

The for loop is used to loop over a sequence. It doesn't take start/stop/step parameters the same way as C, Java, or PERL but you can simulate that.

a = ['rock', 'tree', 'dog', 5, 'F', 34, 'water']
for i in a:
   print(i)

for i in range(0, 25, 2):
    print(i)

range(10)             # stop before 10    
                      # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

range(3, 15)          # start, stop           
                      # [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]

range(0, 10, 2)       # start, stop, step  
                      # [0, 2, 4, 6, 8]

range(-50, -150, -25) # [-50, -75, -100, -125]

a = ['dog', 'cat', 'tree', 'rock', 'house']
for i in range(len(a)):
   print i, a[i]

The enumerate() function can be used to loop over values and their indexes. It returns an enumerate object and can also be used to initialize a list.

a = ['a', 'b', 'c', 'd']

list(enumerate(a))  # return a list of tuples
                    # [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

list(enumerate(a, start=1))         # specify starting index


for i in enumerate(a):              # loop over enumerate object
    print( str(i[0]) + " " + i[1])  # print index and value

To modify a list while looping you should loop over a slice copy of the list. This helps to avoid writing to the same list a that you are looping over. In this example we also use enumerate function so that we can easily index and modify the original list.

a = [5, 1, 2, 9, 4, 5, 7, 3]
for i, x in enumerate(a[:]):  # loop over slice of entire list
    if x % 2:           # odd number
       a[i] += 1        # modify original
print a

For/Else usage

You can use an else statement at the end of a for loop to run something when the loop completes. Breaking out of the loop will skip this.

for x in a:
    print x
else:
    print "done"

Break and continue

• break - break out of current, innermost loop
• continue - continue with next loop iteration

Here is an example of using break to calculate prime numbers. Notice that 'else' pairs with 'for'.

for n in range(2, 25):
   for x in range(2, n):
       if n % x == 0:
           print n, '......'      # not prime
           break                # stop checking this one
   else:
       print n, 'is a PRIME number'

for i in range(2, 25):
   if i % 2 == 0:
       print "Even number", i
       continue
   print "Odd number", i

Pass

Pass is a placeholder. It is silently ignored.

You can use pass for an infinite loop that basically just waits for [ctrl] + [c]:

while True:
   pass  

You can create a valid class or function as a placeholder before implementing anything. Here is an example of a placeholder class:

class MyEmptyClass:
   pass

def function1():
   pass # implement this later...

Functions

• A function returns None if there is no return value.

def function1():
   pass 


function1()        # call it like this

def function1(a, b):
    """Add two numbers."""
    c = a + b
    return c


x = function1(1, 2)  # call it, pass 2 params, return a value
print x              # should be 3 ....


print function1(1, 2)    # can do it like this too

print function1  # just shows the memory address of the function
                 # like this: <function function1 at 0x7f2bc1af7668>


f = function1        # assign to a different name
x = f(1, 2)          # call with the new name
print f              # same memory address 

Function Parameters and Arguments

• Parameter - the parameters defined in the function definition
• Argument - the actual value passed to the function

Function parameters can optionally have default values. Defaults can be overridden. The parameters with defaults don't have to be passed when calling the function. Here is an example:

def silly_user(name, age=0, email='user1@example.com'):
    pass


# only the first param is required
# you can pass the others if you want
silly_user("Fred")                     
silly_user("Jim", 5, "jim123@example.com")

Keyword Arguments

An argument can be specified based on its position or its name. If you mix these, you will need to specify positional arguments first. Keyword arguments can be given in any order so long as they are after the positional arguments.

def test1(a, b, c):
    print a, b, c


test1(1, 2, 3)          # all positional
test1(a=1, b=2, c=3)    # all by name
test1(1, b=2, c=3)      # 1 positional, 2 keyword
test1(1, c=3, b=2)      # order swapped
test1(b=2, c=3, 1)      # ERROR - positional after keyword


def test2(x, y=12, z=5):
    print x, y, z


test2(55)              # 1 positional arg
test2(55, 123)         # 2 positional args
test2(55, 123, 44)     # 3 positional args
test2(x=55)            # 1 keyword arg
test2(x=55, y=123)     # 2 keyword args
test2(y=123, x=55)     # 2 keyword args, out of order
test2(55, y=123)       # 1 positional, 1 keyword 


test2()            # ERROR - missing 1 required arg
test2(y=123, 55)   # ERROR - positional arg after keyword arg
test2(55, x=100)   # ERROR - same arg used twice
test2(55, r=34)    # ERROR - keyword arg doesn't exist

Packing and Unpacking

• *param - this will be a tuple containing all positional args after the formal params
• **param is present - this will be a dictionary containing all keyword args that aren't formal parameters

def test3(a, b, *x, **y):
   print "Formal positional: ", a, b
   for i in x:
       print i
   print "=" * 20
   for m in sorted(y.keys()):
       print m, ":", y[m]


test3('blah', 3, 'one', 'two', w='one', s='two')

def test4( a, b, c):
    print a, b, c


test4(1, 2, 3)             # call it the normal way

args_list = [1, 2, 3]
args_dict = {'a':1, 'b':2, 'c':3}
test4(*args_list)            # unpack args from a list 
test4(**args_dict)           # unpack args from a dict

Edge case example 1:

When a variable is used to set a default value, that assignment takes place at the time of definition. If the source variable is changed later the default value does not change. For example:

i = 24
def test(x=i):      # x has default value of 25
   print x
i = 35              # x doesn't change here
test()              # this will still return 24

Edge case example 2:

x is a mutable object, it accumulates each time:

def test1(a, x=[]):
   x.append(a)
   return x
print test1(1)
print test1(2)
print test1(3)

[1]
[1, 2]
[1, 2, 3]

If you want to initialize this default variable to a list and not have it accumulate you can use the following:

def test1(a, x=None):
   if x is None:
       x = []
   x.append(a)
   return x

print test1(1)
print test1(2)
print test1(3)

Lambda Expressions

• These are small, anonymous functions.
• These can only contain a single expression.

lambda a, b: a+b

f = lambda a, b: a+b     # define
f(1,2)                   # call

You can return a function. In this way one function is used to create another function dynamically.

def create_function(a):
   return lambda x: x + a

f1 = create_function(5)
f2 = create_function(10)
t = f1(8)
r = f2(9)
print t, r

def test1(x):       # takes a lambda as a param
    a = x(5)        # pass 5 to the stored lambda
    a = a + 1       # add 1 just to do something 
    print a         # print the result


test1(lambda i: i * 2)   # pass a lambda that multiplies by 2

This can be used to sort things based on a specific element. The sort function can be passed another function that will decide what to sort on. For example, we might want to sort by the second element rather than the first in a list of tuples. We can just pass a quick lambda that returns that element for each tuple in the list. See the following example:

zoo = [(1, 'frog'), (2, 'bird'), (3, 'zebra'), (4, 'ant')]
zoo.sort(key=lambda e: e[1])       # sort by name
print zoo                          # see them sorted
zoo.sort()        # sort it normally again
print zoo         # see them sorted by number instead

Documentation Strings (docstrings)

Docstrings are important for properly formatted code. See the PEP 8 doc for more details: https://www.python.org/dev/peps/pep-0008/#documentation-strings. An IDE like PyCharm can show PEP 8 violations automatically. The closing line goes on a line by itself unless the entire string is only one line.

def function1():
    """What it does"""
    pass


def function2():
    """What it does 

    Blah blah blah
    More details here.
    """
    pass

Data Structures

• Sequence Types — str, unicode, list, tuple, bytearray, buffer, xrange tuple

Lists as Stacks

• A stack is a LIFO structure ( Last In First Out ).
• A list can be used to implement a stack.

stack = [45, 37, 72]
stack.append(18)        # push onto the stack
x = stack.pop()         # pop off of the stack

Queues

Don't use lists as queues. They are not efficient as queues because popping and appending to the bottom of the list shifts the entire list. Instead use deque.

from collections import deque
queue = deque(["dog", "cat", "monkey"])
queue.append("tiger")          # tiger added to end of queue
queue.append("fish")           # fish added to end of queue
queue.popleft()                # dog popped off queue
queue.popleft()                # cat popped off queue
print queue                    # show the queue

Functional Programming

Filter

This takes two parameters, the first is a function, the second is a sequence. It loops through every element in the sequence passing each one to the function. If the function returns true, that item is added to the return sequence. After going through the entire sequence the filter function will return a sequence. If the sequence passed is a str, unicode, or tuple, the same type will be returned. For anything else, a list is returned.

def odd(x): return x % 2 != 0  # return true if odd
filter(odd, range(25))         # return all odd numbers in range

Map

This takes two parameters, a function and a sequence. It loops through each element in the sequence, passes it to the function, and adds the return value to the return sequence.

def m(x): return x*5  # multiply a value by 5
map(m, range(25))     # return list, each item multiplied by 5

• You can pass multiple sequences, one for each param.
• The corresponding value from each sequence is passed.
• If one sequence is longer, None passed for extra elements. This can cause an error depending on the function.

seq0 = range(25)
seq1 = range(25)
seq2 = range(30)
def add(x, y): return x+y
map(add, seq0, seq1)      # works
map(add, seq1, seq2)      # ERROR - doesn't work, None  
                          # can't be added to an int

Reduce

This takes two parameters, a function and a sequence. the first two elements of the sequence are passed to the function. The result is returned. the next element and the returned result are both passed to the function. This continues for each element in the sequence. returns a single value constructed by calling the binary function function on the first two items of the sequence, then on the result and the next item, and so on. An empty sequence is an error. An single element sequence will just return that one value.

def add(a,b): return a+b
reduce(add, range(25))

A third argument can optionally be passed. This is used as an alternate starting value. It is basically treated as though it were appended to the beginning of the list.

List Comprehensions

List comprehensions are a compact way to process lists. All elements in a lists are looped over, processes, and returned as a new list.

For example, the following will loop over list a, add 1 to each element, and return the results as a new list b.

a = [1, 2, 3, 4, 5]
b = [i + 1 for i in a]

As another example, lets say you want to comput the square of each element in a list. You could do this with a normal for loop or with a list comprehension. See the following:

# using a normal for loop
x = []
for i in range(25):
   x.append(i**2)
print x

# same thing using a list comprehension
x = [i**2 for i in range(25)]
print x

a = [0, 1, 2, 3, 4]
b = [8, 3, 1, 7, 4]


# without list comprehensions
result = []
for x in a:
   for y in b:
       if x != y:
           result.append((x, y))
print result


# using list comprehensions
result = [(x, y) for x in a for y in b if x != y]
print result

result =  [(i, i**2) for i in range(25)]

rock = [[1,2,3], [4,5,6], [7,8,9]]
paper = [h for i in rock for h in i]
print paper

Nested List Comprehensions

You can use nested list comprehensions to transpose the rows and columns in a 2 dimensional list.

grid = [
   [1, 2, 3, 4],
   [5, 6, 7, 8],
   [9, 10, 11, 12],
]


# with nested comprehensions:
result = [[row[i] for row in grid] for i in range(4)]
print result

# without
result = []
for i in range(4):
    temp = []
    for row in grid:
        temp.append(row[i])
    result.append(temp)
print result

Don't actually use this example if you actually need to transpose rows. Use the built-in function zip instead:

zip(*grid)

Tuples

• Immutable
• Usually heterogeneous but don't have to be

t1 = 1234, 56789, 'test'      # define a tuple
t2 = (1234, 56789, 'test')    # define a tuple
t1[0]                         # access an element
print t1                      # print entire tuple

u = t1, (1, 2, 3, 4, 5)       # define nested tuple

t1[0] = 5                     # ERROR - tuples are immutable

v = ([1, 2, 3], [3, 2, 1])    # can contain mutable objects
v[0][0] = 5                   # OK

t1 = ()       # init empty tuple
t2 = 'test',  # init single element tuple (needs trailing comma)
t3 =('test',) # init single element tuple (needs trailing comma)
len(t1)
len(t2)

Packing and unpacking:

t = 01234, 56789, 'test'   # tuple packing
x, y, z = t                # sequence unpacking
a, b = 1, 2                # both - multiple assignment

Sets

• no duplicate elements
• unordered
• can be used for unions, intersections, and other math operations

list1 = ['fig', 'date', 'herring']     # a list ....


set1 = {'fig', 'date', 'herring'}      # init a set
set2 = set(list1)                      # init set with a list
set2 = set(['fig', 'date', 'herring']) # init set with anon list


set3 = set()    # initialize empty set
set4 = {}       # NOT a set!  This is a dict. 


set5 = set('test text, blah')    # initialize with a string

if 'fig' in set2:                # membership check
   print set2

a - b         # elements in a but not in b
a | b         # elements in either a or b
a & b         # elements in both a and b
a ^ b         # elements in a or b but not both

Set comprehensions

a = {i for i in 'text and more' if i not in 'xyz'}
print a

Dictionaries

Dictionaries can also be called associative arrays or hashes. A dict is basically a set of key value pairs. Dictionaries are unordered. Anything can be used as a key so long as it is immutable. Assigning to a new key will create a new element. Assigning to an existing key will overwrite that element.

d = {}                         # empty dictionary
d = {'fig': 84, 'date': 32}    # initialized dictionary

d['fig'] = 87                  # assign value to an element
print d                        # print entire dict
print d['fig']                 # print an element
del d['fig']                   # delete an element
d['salmon'] = 1982             # assign new value

key_list = d.keys()            # get the keys, save as list
key_list = sorted(d.keys())    # same but sorted 

if 'fig' in d:
    print d

The dict() constructor can be used to build a dictionary using key-value pairs. Both of these work:

# pass a list of tuples
d1 = dict([('fig', 12), ('date', 34), ('salmon', 56)])
# pass keyword args
d1 = dict(fig=12, date=34, salmon=56)

Dict Comprehensions

d1 = {x: x**2 for x in (1, 2, 3)}

Looping Techniques

for i, v in enumerate(['frog', 'toad', 'turtle']):
   print i, v

a = ['toast', 'pasta', 'bagel']
b = ['pizza', 'bbq', 'cheese']
for x, y in zip(a, b):
   print x, y

for i in reversed(range(25)):
   print i

a = ['toast', 'pasta', 'bagel']
for i in sorted(set(a)):
   print i

d = {'fig': 84, 'date': 32, 'salmon': 76}
for k, v in d.iteritems():
   print k, v

Quick bit about expressions and conditions

• Expressions are evaluated from left to right.

These are equivalent. The first follows the same order as the second.

x = A and not B or C 
x = (A and (not B)) or C

if a < b == c:
    pass

if x = a:          # ERROR
    pass
if x = a < b:     # ERROR again
    pass

a, b, c = False, True, False
x = a or b or c   
print x               

Comparing Sequences

• For sequences of the same type:
  
  • Compared in order, 1 element at a time, lexicographically
  • First different element is compared and determines outcome
  • Nested sequences are compared recursively
  • If a sequence is a subset of another sequence, it is less than that sequence
• For sequences of different types
  • All lists are smaller than strings, etc.
• For mixed numeric types 0 equals 0.0, etc.

(1, 2, 3)              < (1, 2, 4)
[1, 2, 3]              < [1, 2, 4]
'ABC' < 'C' < 'Pascal' < 'Python'
(1, 2, 3, 4)           < (1, 2, 4)
(1, 2)                 < (1, 2, -1)
(1, 2, 3)             == (1.0, 2.0, 3.0)
(1, 2, ('aa', 'ab'))   < (1, 2, ('abc', 'a'), 4)

More ....

There is a lot more to cover. We're going to continue this in part 2 and part 3 of the tutorial.