python package manager pip

easy_install seems to be out of favor now, here is pip to rescue:

sudo apt-get install python-pip
sudo pip install pip --update
sudo pip install matplotlib
# in many occasions, pip is used to manage environment for projects together with virtualenv
pip install -r requirements.txt -i http://pypi.mine.com

python manipulate matlab .mat file

python:

import scipy.io
# loat matlab .mat into a dictionary
mat = scipy.io.loadmat("mat")
# access variable
print mat["weights34"]
# remove variable
del mat["weights34"]
del mat["bias4"]
# save it
scipy.io.savemat("updated.mat", mat)

scipy provides functionality to read and write matlab .mat files. Loaded .mat is stored as dict in Python. You can add, delete and change as you would do with any dict in python.

python histgram plot with matplotlib

Python:

// install matplotlib with apt-get (Debian, Ubuntu, Mint)
sudo apt-get install python-matplotlib

from matplotlib import pyplot
import random
data = [ random.random() for i in xrange(0, 1000) ]
fig = pyplot.figure(1)
ax = pyplot.subplot(111)
# data is an list of values, use default everything
ax.hist(data)
# most of time, I choose to set bins myself
# ax.hist(data, bins=bins)
ax.set_xlabel("value")
ax.set_xlim(0, 1)
ax.set_ylabel("occuracies")
ax.set_ylim(0, 200)
ax.grid(True)
pyplot.draw()
pyplot.show()

Python: read files in zip format

python:

import zipfile
# "r" is default and can be omitted
zfile = zipfile.ZipFile("files.zip", "r")
# now read each file in zip file
for name in zfile.namelist():
  for line in zfile.open(name):
    print line
zfile.close()

This example is to read an zip archive, to read a single gzip compressed file, please refer to

Python check if a string contains non-utf8 character

Python:

# str.decode and str.encode function can be very useful
try:
  line.decode('UTF-8', 'strict')
except UnicodeDecodeError:
  # line contains non-utf8 character

use fileinput with compressed files

python:

# fileinput is very neat to use
for line in fileinput.input():
  print line
# it is so neat that you can use it with compressed files for free
for line in fileinput.input(openhook=fileinput.hook_compressed):
  print line

python print trace for lines in execution

python:

# be careful with the output, it will flush your screen 😉
python -m trace --trace yourscript.py

“if” in Python lambda function

Python:

# use "and/or" to replace "if/else"
f = lambda x: ( x > 10 and 10 ) or x 

Lambda function is a powerful concept. It helps to make the code clean and clear. Sometimes, the lack of “if/else” can be frustrating. Luckily, with tricks above, you can achieve the same without if/else.

regex with back references followed by number

awk:

awk '{sub("pattern","\\1matched",string);}'

perl:

# replace abcbcd with abcefg, \1 back-reference to matched "abc"
s/(abc)bcd/\1efg/; 
# when there are digits following \1, it can confuse perl, \11 could mean 11th matched group
s/(abc)bcd/\{1}222/;

python:

# replace abcbcd with abcefg, \1 back-reference to matched "abc"
import re
re.sub("(abc)bcd","\1efg","abcbcd");
# when there are digits following \1
re.sub("(abc)bcd","\g<1>222","abcbcd");

randomize lines in a text file

bash:

# if file is not too big
for LINE in `cat text.txt`; do echo "$RANDOM $LINE"; done | sort | sed -r 's/^[0-9]+ //'
cat text.txt | while read LINE; do echo "$RANDOM $LINE"; done | sort | sed -r 's/^[0-9]+ //'
# or as comments cleverly pointed out:
sort -R text.txt
shuf text.txt

python:

# read from standard input and write to standard output for simplicity
import sys
import fileinput
import random

lines = [ line for line in fileinput.input() ]
random.shuffle(lines)
for line in lines:
    print line