Spartan Sniper

Current Status of GSoC Project

2009-07-11T11:07:00.000-07:00

Currently only some tests of execnet/ and test/ failed. The 2nd kind of failures in previous post can be bypassed using locals() dictionary. It seems that python3.1 intendedly suppress the variables generated by exec() can only be shown in locals() dictionary.

Current Status:
tests: 169 failed, 1283 passed.
passed folders: builtin/, code/, cmdline/, io/, log/, misc/, path/, process/, rest/, thread/, tool/, xmlobj/.

Current Status of GSoC project

2009-07-04T10:40:00.000-07:00

Currently, for all tests of py.test, 1277 passed, 183 failed under python3.1. There are mainly 3 kinds of failures:
1. magic/exprinfo.py relies on the compiler package which is no longer supported by python3.x. exprinfo.py is used to construct assertion exception information, and in python3.1 I just simply disable exprinfo.py feature. So, some tests failed because of assertion exception's message does not match the expected string.
2. code/source.py provides a substitute API for compile function, but this API seems does not work well under python3.1. I think the reason is that the compile function's mechanism seems changed. For example:

def foo():
  co = compile("x=3", '', 'exec')
  exec(co)
  print("x:", x)

foo()

The output is: NameError: global name 'x' is not defined. However, this program can be run successfully under python2.x.
3. execnet/ folder aims to support distributed testing feature, and uses io stream tunnel heavily. But, in python3.1, some io streams require bytes not strings. the differences between byte and string are really annoying. So, some tests of py.test failed due to the messages cannot be transported from the sender to the receiver.

Currently passed folders:
builtin/
cmdline/
io/
log/
misc/
path/
process/
rest/
thread/
tool/
xmlobj/

Porting py.test from python2.x to python3.1 (updated on Jun 27)

2009-06-19T20:22:00.000-07:00

Due to python3.x 's intentional backwards incompatibility, it is not an easy work to migrate a project's codebase from python2.x to python3.x. Guido has given a recommended development model:

You should have excellent unit tests with close to full coverage.
Port your project to Python2.6.
Turn on the Py3k warnings mode.
Test and edit until no warnings remain.
Use the 2to3 tool to convert this source code to 3.0 syntax.
Test the converted source code under 3.0.
If problems are found, make corrections to the 2.6 version of the source code and go back to step 4.
When it's time to release, release separate 2.6 and 3.0 tarballs.

Fortunately, py.test is already 2.4 and 2.6 compatible, and has sufficient unit tests. So, I can directly start from step 3. However, unfortunately, Holger requires the codebase ported to 3.1 should still be compatible with 2.4 and 2.6, i.e., maintain only one codebase which could be used through all main python versions. So, 2to3 tool can only be used to show the minimum places that need notification.

Thus, to make py.test compatible with python2.4, 2.6, and 3.1, the first thing is to write some wrapper functions. The functions I wrote are:
Print:
--use Print function instead of each print statement in py.test, so python3.1 will not throw SyntaxError. In Print function, execute correct print code according to current python version.
Raise:
--the only incompatible syntax is raise cls, value, tb (2.x) and raise cls(value).with_traceback(tb) (3.x). So, just simply call corresponding statement in Raise function.
isinstancemethod, isclassmethod, isfunction:
--suppose we defined a class "myclass" and a method "method" inside "myclass", then create an instance of myclass, myinstance. In 2.x, myclass.method and myinstance.method have the same attribute names. But in 3.x, myclass.method's attribute names are the same with the ones of normal function. This is a very annoying difference, because in many places code objects are got by obj.im_func.func_code. In 2.x, obj could be myinstance.method or myclass.method; In 3.x, obj.im_func.func_code must be changed to obj.__func__.__code__ and it only works for myinstance.method. For myclass.method or normal function, should use obj.__code__. So, I have to provide three functions to distinguish them respectively.
updatemethodattr, updatefunctionattr:
--method's attributes:
im_self ==> __self__
im_func ==> __func__
im_class ==> disappeared?
--function's attributes:
func_closure ==> __closure__
func_code ==> __code__
func_defaults ==> __defaults__
func_dict ==> __dict__
func_doc ==> __doc__
func_globals ==> __globals__
func_name ==> __name__
So, to keep codebase unchanged, if obj is 3.x's method or function, create 2.x's attributes.
CmpToKey:
--in 3.x, there is no cmp keyword in sort function. So, this wrapper will transform a cmp function to key function.
bytestostr, strtobytes:
--in 3.x, all strings are unicode, but lots of streams require bytes object instead of string. So, use these two functions to wrap stream arguments and when it's 3.x, transfer from bytes to str or str to bytes.

Second, for some well known incompatibilities, such as "except as" and module rename. For "except Error as e:", change it to "except Error:", and in except block, create e by "e = sys.exc_info()[1]". For module rename, add a try/except block. For example:
try: import StringIO
except ImportError: import io as StringIO

Third, 3.x has many incompatible mechanisms. For example: 3.x will not call __cmp__ when compareing two objects. So, __lt__, __gt__, __le__, __ge__, and __eq__ should be implemented. Furthermore, if you define __eq__, __hash__ must be provided.
In 2.x, dict.items() will return a copy list of key-value pairs. But in 3.x, dict.items() will return a view object, and if the dict's size changed during iteration, a RuntimeError will be thrown. So, although "for key,value in dict.items():" still works in 3.x, it should be changed to "for key, value in list(dict.items()):" in case of RuntimeError.

There are still lots of incompatibities need to be fixed. Currently the packages of py.test without test failures under 3.1 are:
builtin/
cmdline/
io/
log/
path/
process/
rest/
I will keep updating this post when porting. Hope this could be finished ASAP.

Notes on Test

2009-04-23T20:17:00.000-07:00

1. self contained
The test case can be run either in isolation or in arbitrary combination with any number of other test cases.

2. In unittest of Python, If setUp() succeeded, the tearDown() method will be run whether runTest() succeeded or not.

3. In unittest of Python, there are some ways to create test suite.

--------------------
suite = unittest.TestLoader().loadTestsFromTestCase(WidgetTestCase)
--------------------
def suite():
    suite = unittest.TestSuite()
    suite.addTest(WidgetTestCase('testDefaultSize'))
    suite.addTest(WidgetTestCase('testResize'))
    return suite
--------------------
def suite():
    tests = ['testDefaultSize', 'testResize']
    return unittest.TestSuite(map(WidgetTestCase, tests))

4. To use old test without converting every old test function to a TestCase subclass, unittest provides a FunctionTestCase class. This subclass of TestCase can be used to wrap an existing test function. Set-up and tear-down functions can also be provided.

def testSomething():
    something = makeSomething()
    assert something.name is not None
    # ...

testcase = unittest.FunctionTestCase(testSomething)
or
testcase = unittest.FunctionTestCase(testSomething,
setUp=makeSomethingDB,
tearDown=deleteSomethingDB)

5. TestCase.setUp()
This is called immediately before calling the test method; any exception raised by this method will be considered an error rather than a test failure. The default implementation does nothing.

6. The methods of TestCase used by the test implementation to check conditions and report failures can be get here.

7. ad-hoc means that there are no installation requirements whatsoever on the remote side. (from here)

8. By default, py.test catches text written to stdout/stderr during the execution of each individual test. This output will only be displayed however if the test fails, each failing test that produced output during the running of the test will have its output displayed in the recorded stdout section.

9. Disabling a test class
If you want to disable a complete test class you can set the class-level attribute disabled. For example, in order to avoid running some tests on Win32:

class TestPosixOnly:
    disabled = sys.platform == 'win32'
    def test_xxx(self):
        ...

GSoC Application Accepted

2009-04-20T14:47:00.000-07:00

It is wonderful for me to know that my application was accepted by Google Summer of Code program (GSoC). This summer I will work for improving py.test compatibility with several versions of Python, especially 3k. Besides, better-formatted output of regression tests and code coverage will be shown regularly on Snakebite. This project, accepted into Python Software Foundation (PSF), was inspired by Titus Brown, and will be mentored by Holger Krekel. Titus and Holger have given me many precious advices and suggestions, which are greatly appreciated.

There is no need to keep my proposal from public view, for it is against the spirit of open source. Thus, I paste it as following, and all comments, advices, or suggestions are welcome.

----GSoC Application Proposal----

ABOUT ME
My name is Yang Yang, a Ph.D. student in Computer Science at Michigan State University. As a teaching assistant of a Python course for nearly two years, I found that Python is such an amazing programming language that freshman can learn programming concepts effectively without being trapped in lower level details, meanwhile it is very powerful and can be used in almost everywhere. I am using Python almost every day, ranging from my research work to courses projects. Compared with C++, Python could save me lots of time in programming, thanks to its fewer alternatives and elegant coding style.

I still remember the time when we were developing a P2P-VoD application using C++. We have to write our own trace function and every one took charge of testing his/her own modules. Lacking of testing experience, we had to spend excessive time on debugging. Thus, knowing that py.test is aiming to provide a powerful testing framework, I am eager to improve this tool to make it usable for Core Python3k developers. So, developers can run tests without thought and more concentrate on their own modules.

This project will be my main work in this summer, and I am able to be in contact with mentors and community. Furthermore, finishing this project does not mean I already reach the destination. I will still keep in contact within PSF community and keep making my contribution.

CONTACT INFO
email: yangyan5 AT msu DOT edu
yangyan5 AT gmail DOT com
cell: 517-614-6078
IRC nick: yangyang

TITLE
Improve Unit Testing Framework for Core Python3k

ABSTRACT
In this project, I will make py.test compatible to Core Python3k test files and introduce some features to py.test, such that Python developers can run unit testing easier. Besides, regression test suite will be run over multiple platforms regularly on Snakebite, integrated with build-bot or other facilities.

SUMMARY
Unit testing is very important in software development and Python has already provided a unit testing framework unittest. Although unittest supports some features such as test automation and collection, tests still cannot be run without thought, which means that we still need to register the tests we want to run. Thus, some unit testing framework arises, such as py.test and nose. py.test is already used in PyPy and it has many excellent features, such as automatically collecting and executing tests, selecting/unselecting tests by keyword, and distributing tests to multiple CPUs or machines. Similarly, nose provides a number of excellent features as well. For example, nose has the ability to run tests based on specific tags, i.e. tests can be run selectively without manually generating test suite. Besides, nose can transparently wrap tests with code coverage recording. However, the prerequisite of using these features of py.test and nose is that the tests must be discoverable and executable, which cannot be guaranteed in Core Python. Besides, py.test is still not compatible with Python3k. Thus, I plan to introduce several features that make py.test and nose compatible and convenient to run the hundreds of test files in Core Python.

My effort will mainly focus on Python3k and be seven-fold. The first feature I'd like to contribute is to make py.test compatible with Python3k as well as Python2.4, which will be the basis of my later work. Second, I will provide a py.test plug-in for Python3k, Python2.6, and Python2.5 that could "understand" the conditions under which regrtest.py, the current test runner, finds and runs tests. Third, as Jython is becoming more and more popular, I plan to port the plug-in to make py.test could be used in Jython. Fourth, I will set up regular test runs for Core Python3k on Snakebite such that regression test suite across multiple platforms can be recorded and posted. And, for daily run reporting on Snakebite, I will work on integrating with build-bot or other facilities. Fifth, to help developers test their code over different version of Python, I will check the latest features of unittest.py and doctest.py of Python3k and make them compatible with previous versions of Python. Sixth, aiming to increase the test coverage in Python3k, I plan to write some test cases and report current test coverage for python modules in stdlib and list achieved improvements. Although py.test has pytest_figleaf which offers coverage report, the report is raw and needs configuration to make it useful and meaningful for developers. Last but not least, to collaborate with nose, a plug-in for py.test will be provided to invoke and run nose-style tests, which means to make py.test could discover and execute nose-style tests.

Through all of my work, py.test will be made compatible with multiple versions of CPython and Jython when running regression test suite, without modifying tests too much. In addition, the regression tests and code coverage result over multiple platforms will be shown on Snakebite.

SCHEDULE
April 20 - May 22: Get to know mentors and community. Dive into the details of py.test, regrtest.py, and nose.
May 23 - June 7: Port py.test to Python3k, keeping compatible with Python2.4 as well.. (including debugging)
June 8 - June 21: Write a Plug-in to make py.test compatible with regrtest of Python3k, Python2.6, and Python2.5. (including debugging)
June 22 - June 28: Port py.test and plug-in to Jython. (including debugging)
June 29 - July 5: Port latest features of Python3k to previous versions. (including debugging)
July 6 - July 15: Set up regular regression test runs on Snakebite. (keeping in touch with python-dev)
July 16 - July 26: Write test cases, report current test coverage and list achieved improvements. (keeping in touch with python-dev)
July 27 - August 2: Write a plug-in for py.test to invoke/run nose-style tests.
August 3 - August 10: Documentation

DELIVERABLES
1. Port py.test to Python3k, keeping compatible with Python2.4 as well.
2. A plug-in to make py.test compatible with regrtest of Python3k, Python2.6, and Python2.5.
3. Port py.test and plug-in to Jython.
4. Port latest features of Python3k to previous versions.
5. Regression test suite regular report on Snakebite, integrated with build-bot or other facilities.
6. Report current test coverage and list achieved improvements. The report will be developer-friendly.
7. A plug-in for py.test to invoke/run nose-style tests.

BENEFITS TO COMMUNITY
1. Make unit testing on Core Python3k and other Python versions easier by allowing developers to use py.test with some convenient features.
2. Give developers a clear view on regression test and code coverage result over multiple platforms.
-------End of Proposal--------

The exciting and busy summer is coming!

Notes on Mercurial

2009-04-19T22:25:00.000-07:00

Mercurial is a distributed source version control system. Compared with subversion, Mercurial is much more scalable, and everyone can commit his/her own effort. It is becoming more and more popular, and Python has switched to Mercurial recently. Here is a tutorial written by Bryan O'Sullivan and all notes below are extracted from it.

1. log
$ hg log -r [-v] [-p]
-v: verbose mode
-p: show content of a change

2. status
$ hg status
Show what Mercurial knows about the files in repository.

3. incoming, pull, update
We use pull to get the changesets from "remote" repository. But, to avoid blindly fetching, we should use incoming first to give us a clear view of what will be transferred.
$ hg incoming remote_repository
$ hg pull remote_repository
Because pull only bring changes into repository, we should use update to get a working copy. So, we should run:
$ hg update

4. outgoing, push, update
outgoing command will tell us what changes would be pushed into another repository. push command does the actual push. Then use update to get a working copy.
$ hg outgoing remote_repository
$ hg push remote_repository
$ hg update

5. heads, parents
We can view the heads in a repository using the heads command.
$ hg heads
Note: heads is different with parents. parents command is used to find out what revision the working directory is at.
$ hg parents

6. Removing a file does not affect its history. It has only two effects:

It removes the current version of the file from the working directory.
It stops Mercurial from tracking changes to the file, from the time of the next commit.

7. Mercurial considers a file that you have deleted, but not used hg remove to delete, to be missing. If you deleted the missing file by accident, give hg revert the name of the file to recover. It will reappear, in unmodified form.

8. Mercurial offers a combination command, hg addremove, that adds untracked files and marks missing files as removed.

An interesting post "Programmer Insecurity"

2009-04-17T11:08:00.000-07:00

Holger Krekel shared with me an interesting post which talks about some typical prototypes of programmers. It is so true that I found those characteristics in myself and teammates very often. For example, I thought it embarrassing me to let others see my iterative work before I reach a milestone. And the result is that I have no time to refactor my code and have to continue my work based on the fragile and awkward "code bomb". Thanks to that post, I am trying to put the advices into practice.

Notes on Python builtin functions

2009-04-09T10:08:00.000-07:00

Recently I have been shifting from Python2.x to Python3.x. So, all notes below are only guaranteed in Python3.x.

1. all(iterable) and any(iterable)
They are equivalent to following functions respectively:

def all(iterable):
for element in iterable:
 if not element:
     return False
return True

def any(iterable):
for element in iterable:
 if element:
     return True
return False

2. bin(number) and hex(number)
Convert an integer number to a binary or hexadecimal string (starts with '0b' or '0x'). The result is a valid Python expression.

3. chr(i)
Return the string of one character whose Unicode codepoint is the integer i. For example, chr(97) returns the string 'a'. This is the inverse of ord().

4. classmethod
The @classmethod form is a function decorator. A class method receives the class as implicit first argument, just like an instance method receives the instance. To declare a class method, use this idiom:

class C:
@classmethod
def f(cls, arg1, arg2, ...): ...

It can be called either on the class (such as C.f()) or on an instance (such as C().f()). The instance is ignored except for its class. If a class method is called for a derived class, the derived class object is passed as the implied first argument. Note that class methods are different than C++ or Java static methods.

5. delattr(object, name)
This is a relative of setattr(). The arguments are an object and a string. The string must be the name of one of the object’s attributes. The function deletes the named attribute, provided the object allows it.

6. dir([object])
Without arguments, return the list of names in the current local scope. With an argument, attempt to return a list of valid attributes for that object.

7. divmod(a, b)
Take two (non complex) numbers as arguments and return a pair of numbers consisting of their quotient and remainder.

8. enumerate(iterable[, start=0])
Return a tuple containing a count (from start which defaults to 0) and the corresponding value obtained from iterating over iterable.

9. eval(expression[, globals[, locals]]), exec(object[, globals[, locals]]), and compile(source, filename, mode[, flags[, dont_inherit]])
Here I just show some simple examples. For more details, please check here.

>>> glo = {'x':5, 'y':6}
>>> eval('x+y', glo)
11

>>> glo = {'aList':[1,2,3]}
>>> exec('for i in aList: print i', glo)
1
2
3

>>> aString = 'for i in range(3): print i'
>>> aStatement = compile(aString, '', 'exec')
>>> exec(aStatement)
0
1
2

Note:
a) The built-in functions globals() and locals() return the current global and local dictionary, respectively, which may be useful to pass around for use as the second and third argument to exec().
b) execfile function is no longer used in Python3.x. Instead of execfile(fn) use exec(open(fn).read()).

10. frozenset([iterable])
To represent sets of sets, the inner sets must be frozenset objects.

11. getattr(object, name[, default])
Return the value of the named attributed of object. name must be a string. If the string is the name of one of the object’s attributes, the result is the value of that attribute. For example, getattr(x, 'foobar') is equivalent to x.foobar.

12. hasattr(object, name)
The arguments are an object and a string. The result is True if the string is the name of one of the object’s attributes, False if not. (This is implemented by calling getattr(object, name) and seeing whether it raises an exception or not.)

13. hash(object)
Return the hash value of the object (if it has one). Hash values are integers. They are used to quickly compare dictionary keys during a dictionary lookup. Numeric values that compare equal have the same hash value (even if they are of different types, as is the case for 1 and 1.0).

14. id(object)
Return the “identity” of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same id value. (Implementation note: this is the address of the object.)

15. memoryview(obj)
Return a “memory view” object created from the given argument. For example:

>>> data = bytearray(b'abcefg')
>>> v = memoryview(data)
>>> v.readonly
False
>>> v[0] = 'z'
>>> data
bytearray(b'zbcefg')
>>> v[1:4] = b'123'
>>> data
bytearray(b'a123fg')
>>> v[2] = b'spam'
Traceback (most recent call last):
File "", line 1, in 
ValueError: cannot modify size of memoryview object

There is a tolist() method which returns the data in the buffer as a list of integers:

>>> memoryview(b'abc').tolist()
[97, 98, 99]

16. setattr(object, name, value)

This is the counterpart of getattr(). The arguments are an object, a string and an arbitrary value. The string may name an existing attribute or a new attribute. The function assigns the value to the attribute, provided the object allows it.

17. staticmethod

A static method does not receive an implicit first argument. To declare a static method, use this idiom:

class C:
 @staticmethod
 def f(arg1, arg2, ...): ...

It can be called either on the class (such as C.f()) or on an instance (such as C().f()). The instance is ignored except for its class.

18. vars([object])
Without arguments, return a dictionary corresponding to the current local symbol table. With a module, class or class instance object as argument (or anything else that has a __dict__ attribute), returns a dictionary corresponding to the object’s symbol table.

An annoying error

2009-03-30T14:47:00.000-07:00

Today I was trying to run nose on some tests, but keeping getting this error:

'import site' failed; use -v for traceback

At first I doubted that the problem was derived from two different versions of Python. A few days ago I downloaded and built python2.7 which is still in the trunk. When I built it, I forgot to specify the tag --prefix. So, python2.7 was installed into /usr/local/, and the python command was directed to python2.7. However, when I built nose, it was extracted into /usr/lib/python2.5/site-packages/. Thus, nose kept giving me the error above. So, I removed python2.7 (this step took me a long time because I was not familiar with the locations where python2.7 was installed.). Guess what, I still could not run nose. I tried "python -V" to get the current version, and found that the current version was still python2.7. It turned out that I forgot to delete the binary file "python" under /usr/local/bin/. OK, I directly copy the binary file under /usr/bin/ to overwrite the one in /usr/local/bin/. And still, nose did not work. I got nearly crazy and came near to reinstall the operating system. Finally, I added two lines into .bashrc file:

PYTHONPATH="/usr/lib/python2.5"
PYTHONHOME="/usr/lib/python2.5"

and then rebooted the operating system (I tried source command but nose still could not work.). Thank God! Everything works fine now.

Notes & tips in Python

2009-03-14T14:02:00.000-07:00

If you really want to dive into a programming language, the best way is to read its docs. Thus, all notes below are fetched from Python2.6 Doc.
1. Assignment to slices is also possible, and this can even change the size of the list or clear it entirely:

>>> # Replace some items:
... a[0:2] = [1, 12]
>>> a
[1, 12, 123, 1234]
>>> # Remove some:
... a[0:2] = []
>>> a
[123, 1234]
>>> # Insert some:
... a[1:1] = ['bletch', 'xyzzy']
>>> a
[123, 'bletch', 'xyzzy', 1234]
>>> # Insert (a copy of) itself at the beginning
>>> a[:0] = a
>>> a
[123, 'bletch', 'xyzzy', 1234, 123, 'bletch', 'xyzzy', 1234]
>>> # Clear the list: replace all items with an empty list
>>> a[:] = []
>>> a
[]

2. It is not safe to modify the sequence being iterated over in the loop (this can only happen for mutable sequence types, such as lists). If you need to modify the list you are iterating over (for example, to duplicate selected items) you must iterate over a copy. The slice notation makes this particularly convenient:

>>> a = ['cat', 'window', 'defenestrate']
>>> for x in a[:]: # make a slice copy of the entire list
...    if len(x) > 6: a.insert(0, x)
...
>>> a
['defenestrate', 'cat', 'window', 'defenestrate']

3. An example to show the slice of list:

>>> a = [1,2,3]
>>> a[:] = []
>>> a
[]
>>> a = [1,2,3]
>>> b = a
>>> b.append(4)
>>> b
[1,2,3,4]
>>> a
[1,2,3,4]
>>> c = a[:]
>>> c.append(5)
>>> c
[1,2,3,4,5]
>>> a
[1,2,3,4]

4. Loop statements may have an else clause; it is executed when the loop terminates through exhaustion of the list (with for) or when the condition becomes false (with while), but not when the loop is terminated by a break statement.

5. A function definition introduces the function name in the current symbol table. The value of the function name has a type that is recognized by the interpreter as a user-defined function. This value can be assigned to another name which can then also be used as a function.

6. The default values are evaluated at the point of function definition in the defining scope, so that:

i = 5

def f(arg=i):
print arg

i = 6
f()

will print 5. The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes. For example, the following function accumulates the arguments passed to it on subsequent calls:

def f(a, L=[]):
L.append(a)
return L

print f(1)
print f(2)
print f(3)

This will print:

[1]
[1, 2]
[1, 2, 3]

If you don’t want the default to be shared between subsequent calls, you can write the function like this instead:

def f(a, L=None):
if L is None:
    L = []
L.append(a)
return L

7. When the arguments are already in a list or tuple, it can be unpacked for a function call requiring separate positional arguments.

>>> range(3, 6)
[3, 4, 5]
>>> args = [3, 6]
>>> range(*args)
[3, 4, 5]

8. For Python, PEP 8 has emerged as the style guide that most projects adhere to; it promotes a very readable and eye-pleasing coding style. Every Python developer should read it at some point; I just extract some points here:

When possible, put comments on a line of their own.
Use docstrings.
Name your classes and functions consistently; the convention is to use CamelCase for classes and lower_case_with_underscores for functions and methods.
Don’t use fancy encodings if your code is meant to be used in international environments. Plain ASCII works best in any case.

9. list.insert(i, x)
Insert an item at a given position. The first argument is the index of the element before which to insert, so a.insert(0, x) inserts at the front of the list, and a.insert(len(a), x) is equivalent to a.append(x).

10. There are three built-in functions that are very useful when used with lists: filter(), map(), and reduce().
filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true. If sequence is a string or tuple, the result will be of the same type; otherwise, it is always a list. For example, to compute some primes:

>>> def f(x): return x % 2 != 0 and x % 3 != 0
...
>>> filter(f, range(2, 25))
[5, 7, 11, 13, 17, 19, 23]

map(function, sequence) calls function(item) for each of the sequence’s items and returns a list of the return values. For example, to compute some cubes:

>>> def cube(x): return x*x*x
...
>>> map(cube, range(1, 11))
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]

More than one sequence may be passed; the function must then have as many arguments as there are sequences and is called with the corresponding item from each sequence (or None if some sequence is shorter than another). For example:

>>> seq = range(8)
>>> def add(x, y): return x+y
...
>>> map(add, seq, seq)
[0, 2, 4, 6, 8, 10, 12, 14]

reduce(function, sequence) returns a single value constructed by calling the binary function function on the first two items of the sequence, then on the result and the next item, and so on. For example, to compute the sum of the numbers 1 through 10:

>>> def add(x,y): return x+y
...
>>> reduce(add, range(1, 11))
55

11. Consider the following example of a 3x3 matrix held as a list containing three lists, one list per row:

>>> mat = [
...        [1, 2, 3],
...        [4, 5, 6],
...        [7, 8, 9],
...       ]

Now, if you wanted to swap rows and columns, you could use a list comprehension:

>>> print [[row[i] for row in mat] for i in [0, 1, 2]]
[[1, 4, 7], [2, 5, 8], [3, 6, 9]]

In real world, you should prefer builtin functions to complex flow statements. The zip() function would do a great job for this use case:

>>> zip(*mat)
[(1, 4, 7), (2, 5, 8), (3, 6, 9)]

12. the del statement differs from the pop() method which returns a value. The del statement can also be used to remove slices from a list or clear the entire list (which we did earlier by assignment of an empty list to the slice). For example:

>>> a = [-1, 1, 66.25, 333, 333, 1234.5]
>>> del a[0]
>>> a
[1, 66.25, 333, 333, 1234.5]
>>> del a[2:4]
>>> a
[1, 66.25, 1234.5]
>>> del a[:]
>>> a
[]

del can also be used to delete entire variables:

>>> del a

Referencing the name a hereafter is an error (at least until another value is assigned to it). We’ll find other uses for del later.

13. Here are some set operations:

>>> a = set('abracadabra')
>>> b = set('alacazam')
>>> a      # unique letters in a
set(['a', 'r', 'b', 'c', 'd'])
>>> a - b  # letters in a but not in b
set(['r', 'd', 'b'])
>>> a  b   # letters in either a or b
set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
>>> a & b  # letters in both a and b
set(['a', 'c'])
>>> a ^ b  # letters in a or b but not both
set(['r', 'd', 'b', 'm', 'z', 'l'])

14. enumerate function will return a list of tuples. The first element in each tuple is the index. For example:

>>> a = [1,2,3,4]
>>> for x in enumerate(a):
...     print x
(0, 1)
(1, 2)
(2, 3)
(3, 4)
>>> for x, y in enumerate(a):
...     print x, y
0 1
1 2
2 3
3 4

15. To loop over a sequence in reverse, first specify the sequence in a forward direction and then call the reversed() function. To loop over a sequence in sorted order, use the sorted() function which returns a new sorted list while leaving the source unaltered.

16. in Python, unlike C, assignment cannot occur inside expressions. C programmers may grumble about this, but it avoids a common class of problems encountered in C programs: typing = in an expression when == was intended.

17.

from modulename import *

This imports all names except those beginning with an underscore (_).

18. Modules are searched in current directory and the list of directories given by the variable sys.path.

19. A program doesn’t run any faster when it is read from a .pyc or .pyo file than when it is read from a .py file; the only thing that’s faster about .pyc or .pyo files is the speed with which they are loaded.

20. It is possible to have a file called spam.pyc (or spam.pyo when -O is used) without a file spam.py for the same module. This can be used to distribute a library of Python code in a form that is moderately hard to reverse engineer. The module compileall can create .pyc files (or .pyo files when -O is used) for all modules in a directory.

21. The variables sys.ps1 and sys.ps2 define the strings used as primary and secondary prompts:

>>> import sys
>>> sys.ps1
'>>> '
>>> sys.ps2
'... '
>>> sys.ps1 = 'C> '
C> print 'Yuck!'
Yuck!
C>

These two variables are only defined if the interpreter is in interactive mode.

22. Without arguments, dir() lists the names you have defined currently:

>>> a = [1, 2, 3, 4, 5]
>>> import fibo
>>> fib = fibo.fib
>>> dir()
['__builtins__', '__doc__', '__file__', '__name__', 'a', 'fib', 'fibo', 'sys']

Note that it lists all types of names: variables, modules, functions, etc. dir() does not list the names of built-in functions and variables. If you want a list of those, they are defined in the standard module __builtin__.

23. Note that when using from package import item, the item can be either a submodule (or subpackage) of the package, or some other name defined in the package, like a function, class or variable. The import statement first tests whether the item is defined in the package; if not, it assumes it is a module and attempts to load it. If it fails to find it, an ImportError exception is raised.

24. The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package.

25. There is another method, zfill(), which pads a numeric string on the left with zeros. It understands about plus and minus signs:

>>> '12'.zfill(5)
'00012'
>>> '-3.14'.zfill(7)
'-003.14'
>>> '3.14159265359'.zfill(5)
'3.14159265359'

26. If the end of the file has been reached, f.read() will return an empty string ("").

>>> f.read()
'This is the entire file.\n'
>>> f.read()
''

27. f.tell() returns an integer giving the file object’s current position in the file, measured in bytes from the beginning of the file. To change the file object’s position, use f.seek(offset, from_what). The position is computed from adding offset to a reference point; the reference point is selected by the from_what argument. A from_what value of 0 measures from the beginning of the file, 1 uses the current file position, and 2 uses the end of the file as the reference point. from_what can be omitted and defaults to 0, using the beginning of the file as the reference point.

>>> f = open('/tmp/workfile', 'r+')
>>> f.write('0123456789abcdef')
>>> f.seek(5)     # Go to the 6th byte in the file
>>> f.read(1)
'5'
>>> f.seek(-3, 2) # Go to the 3rd byte before the end
>>> f.read(1)
'd'

28. It is good practice to use the with keyword when dealing with file objects. This has the advantage that the file is properly closed after its suite finishes, even if an exception is raised on the way. It is also much shorter than writing equivalent try-finally blocks:

>>> with open('/tmp/workfile', 'r') as f:
...     read_data = f.read()
>>> f.closed
True

29. Python provides a standard module called pickle. This is an amazing module that can take almost any Python object (even some forms of Python code!), and convert it to a string representation; this process is called pickling. Reconstructing the object from the string representation is called unpickling. Between pickling and unpickling, the string representing the object may have been stored in a file or data, or sent over a network connection to some distant machine.
If you have an object x, and a file object f that’s been opened for writing, the simplest way to pickle the object takes only one line of code:

pickle.dump(x, f)

To unpickle the object again, if f is a file object which has been opened for reading:

x = pickle.load(f)

30. A good example to show try-except block:

import sys

try:
   f = open('myfile.txt')
   s = f.readline()
   i = int(s.strip())
except IOError as (errno, strerror):
   print "I/O error({0}): {1}".format(errno, strerror)
except ValueError:
   print "Could not convert data to an integer."
except:
   print "Unexpected error:", sys.exc_info()[0]
   raise

31. In real world applications, the finally clause is useful for releasing external resources (such as files or network connections), regardless of whether the use of the resource was successful.

32. Name spaces are created at different moments and have different lifetimes. The namespace containing the built-in names is created when the Python interpreter starts up, and is never deleted. The global namespace for a module is created when the module definition is read in; normally, module namespaces also last until the interpreter quits. The statements executed by the top-level invocation of the interpreter, either read from a script file or interactively, are considered part of a module called __main__, so they have their own global namespace.(The built-in names actually also live in a module; this is called __builtin__.)

33. In class, Data attributes override method attributes with the same name.

34. Derived classes may override methods of their base classes. Because methods have no special privileges when calling other methods of the same object, a method of a base class that calls another method defined in the same base class may end up calling a method of a derived class that overrides it. (For C++ programmers: all methods in Python are effectively virtual.)

35. Python has two builtin functions that work with inheritance:

Use isinstance() to check an object’s type: isinstance(obj, int) will be True only if obj.__class__ is int or some class derived from int.
Use issubclass() to check class inheritance: issubclass(bool, int) is True since bool is a subclass of int. However, issubclass(unicode, str) is False since unicode is not a subclass of str (they only share a common ancestor, basestring).

36. Instance method objects have attributes, too: m.im_self is the instance object with the method m(), and m.im_func is the function object corresponding to the method.

37. There are two new valid (semantic) forms for the raise statement:

raise Class, instance

raise instance

In the first form, instance must be an instance of Class or of a class derived from it. The second form is a shorthand for:

raise instance.__class__, instance

38. To add iterator behavior to your classes, define a __iter__() method which returns an object with a next() method. If the class defines next(), then __iter__() can just return self:

class Reverse:
   "Iterator for looping over a sequence backwards"
   def __init__(self, data):
       self.data = data
       self.index = len(data)
   def __iter__(self):
       return self
   def next(self):
       if self.index == 0:
           raise StopIteration
       self.index = self.index - 1
       return self.data[self.index]

>>> for char in Reverse('spam'):
...     print char
...
m
a
p
s

Generators are a simple and powerful tool for creating iterators. They are written like regular functions but use the yield statement whenever they want to return data. Each time next() is called, the generator resumes where it left-off (it remembers all the data values and which statement was last executed). An example shows that generators can be trivially easy to create:

def reverse(data):
   for index in range(len(data)-1, -1, -1):
       yield data[index]

>>> for char in reverse('golf'):
...     print char
...
f
l
o
g

The yield statement is only used when defining a generator function, and is only used in the body of the generator function. Using a yield statement in a function definition is sufficient to cause that definition to create a generator function instead of a normal function. When a generator function is called, it returns an iterator known as a generator iterator, or more commonly, a generator. The body of the generator function is executed by calling the generator’s next() method repeatedly until it raises an exception. When a yield statement is executed, the state of the generator is frozen and the value of expression_list is returned to next()'s caller. By "frozen" we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, and the internal evaluation stack: enough information is saved so that the next time next() is invoked, the function can proceed exactly as if the yield statement were just another external call.

39. Be sure to use the import os style instead of from os import *. This will keep os.open() from shadowing the builtin open() function which operates much differently.

40. The glob module provides a function for making file lists from directory wildcard searches:

>>> import glob
>>> glob.glob("C:\\Users\\Yang\\Desktop\\*.py")
['C:\\Users\\Yang\\Desktop\\file_preprocess.py', 'C:\\Users\\Yang\\Desktop\\htmlFunctions.py', 'C:\\Users\\Yang\\Desktop\\proj03.py', 'C:\\Users\\Yang\\Desktop\\template.py', 'C:\\Users\\Yang\\Desktop\\test.py']

41. The most direct way to terminate a script is to use sys.exit().

42. The datetime module supplies classes for manipulating dates and times in both simple and complex ways.

# dates are easily constructed and formatted
>>> from datetime import date
>>> now = date.today()
>>> now
datetime.date(2003, 12, 2)
>>> now.strftime("%m-%d-%y. %d %b %Y is a %A on the %d day of %B.")
'12-02-03. 02 Dec 2003 is a Tuesday on the 02 day of December.'

# dates support calendar arithmetic
>>> birthday = date(1964, 7, 31)
>>> age = now - birthday
>>> age.days
14368

43. The textwrap module formats paragraphs of text to fit a given screen width:

>>> import textwrap
>>> doc = """The wrap() method is just like fill() except that it returns
... a list of strings instead of one big string with newlines to separate
... the wrapped lines."""
...
>>> print textwrap.fill(doc, width=40)
The wrap() method is just like fill()
except that it returns a list of strings
instead of one big string with newlines
to separate the wrapped lines.

44. The locale module accesses a database of culture specific data formats. The grouping attribute of locale’s format function provides a direct way of formatting numbers with group separators:

>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
'English_United States.1252'
>>> conv = locale.localeconv() # get a mapping of conventions
>>> x = 1234567.8
>>> locale.format("%d", x, grouping=True)
'1,234,567'
>>> locale.format("%s%.*f", (conv['currency_symbol'],
...        conv['frac_digits'], x), grouping=True)
'$1,234,567.80'

45. The following code shows how the high level threading module can run tasks in background while the main program continues to run:

import threading, zipfile

class AsyncZip(threading.Thread):
   def __init__(self, infile, outfile):
       threading.Thread.__init__(self)
       self.infile = infile
       self.outfile = outfile
   def run(self):
       f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
       f.write(self.infile)
       f.close()
       print 'Finished background zip of: ', self.infile

background = AsyncZip('mydata.txt', 'myarchive.zip')
background.start()
print 'The main program continues to run in foreground.'

background.join() # Wait for the background task to finish
print 'Main program waited until background was done.'

While those tools are powerful, minor design errors can result in problems that are difficult to reproduce. So, the preferred approach to task coordination is to concentrate all access to a resource in a single thread and then use the Queue module to feed that thread with requests from other threads. Applications using Queue.Queue objects for inter-thread communication and coordination are easier to design, more readable, and more reliable.

46. The array module provides an array() object that is like a list that stores only homogeneous data and stores it more compactly. The following example shows an array of numbers stored as two byte unsigned binary numbers (typecode "H") rather than the usual 16 bytes per entry for regular lists of python int objects:

>>> from array import array
>>> a = array('H', [4000, 10, 700, 22222])
>>> sum(a)
26932
>>> a[1:3]
array('H', [10, 700])

47. The collections module provides a deque() object that is like a list with faster appends and pops from the left side but slower lookups in the middle. These objects are well suited for implementing queues and breadth first tree searches:

>>> from collections import deque
>>> d = deque(["task1", "task2", "task3"])
>>> d.append("task4")
>>> print "Handling", d.popleft()
Handling task1

unsearched = deque([starting_node])
def breadth_first_search(unsearched):
   node = unsearched.popleft()
   for m in gen_moves(node):
       if is_goal(m):
           return m
       unsearched.append(m)

48. The heapq module provides functions for implementing heaps based on regular lists. The lowest valued entry is always kept at position zero. This is useful for applications which repeatedly access the smallest element but do not want to run a full list sort:

>>> from heapq import heapify, heappop, heappush
>>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
>>> heapify(data)                      # rearrange the list into heap order
>>> heappush(data, -5)                 # add a new entry
>>> [heappop(data) for i in range(3)]  # fetch the three smallest entries
[-5, 0, 1]

49. Non-identical instances of a class normally compare as non-equal unless the class defines the __eq__() method.

50. int.bit_length() returns the number of bits necessary to represent an integer in binary, excluding the sign and leading zeros:

>>> n = -37
>>> bin(n)
'-0b100101'
>>> n.bit_length()
6

51. str.partition(sep)
Split the string at the first occurrence of sep, and return a 3-tuple containing the part before the separator, the separator itself, and the part after the separator. If the separator is not found, return a 3-tuple containing the string itself, followed by two empty strings.

Why my blog's name is Spartan Sniper?

2008-12-04T06:33:00.000-08:00

Sparty is the mascot of Michigan State University, and we call us spartans, which shows strong laconophilia. For sniper, I am a fan of Counter-Strike, and I am very good at using AWP. Hmmm, well I was. But I still think AWP is the most beautiful weapon. That's why I pick Spartan Sniper as my blog's name.

The first post on this blog

2008-12-04T06:00:00.000-08:00

Hello, This is Yang. I am a Ph.D. student in Computer Science and Engineering Department of Michigan State University. Google provides so many interesting stuff, so I think creating a blog on Google can take advantage of these stuff seamlessly. This is my first blog in English, which blocks me very much to express my train of thought effectively. So I don't plan to write a novel or poem here except some sketches of what I am interested in or what I encountered. Thank you all and Merry Christmas. God bless the world.