1) Introduction

• This course has introduced you to the skills and tools that differentiate productive programmers from unproductive ones
• But we've really just scratched the surface
• This lecture looks at a few of the next steps you might want to take

2) Classic Mistakes

• People
  • Adding people to a late project (just makes it later [Brooks 1995])
  • Relying on heroics (don't scale [Robinson 2006])
  • Lack of support from project sponsors
  • Lack of user input
  • Silver bullet syndrome
• Product
  • Gold-plating
  • Feature creep
• Process
  • Overly optimistic schedules
    • Remember, products take three times longer to create than programs
  • Short-changing upstream activities
    • "We have to start coding right away, or we won't have time to fix all our bugs."
  • Failure to track progress
  • Abandoning plan under pressure

3) Branching, Merging, and Tagging

• Sometimes want to work on several different versions of software at once
  • Example: need to do bug fixes on Version 3 while making incompatible changes toward Version 4
  • Or want two sets of developers to be able to write and test large changes independently, then put things back together
• All modern version control systems allow you to branch a repository
  • Create a "parallel universe" which is initially the same as the original, but which evolves independently
  • Can later merge changes from one branch to another
  

  Figure 27.1: Branching and Merging

• Also common to create tags
  • Symbolic labels that identify particular revisions, such as "Release_2.0"
  • Makes it easy to go back to an important revision later

4) Managing Branches

• Much better than just copying all the source files
  • The version control system remembers where the branch came from, and can trace its history back
  • Example: fix a bug on one branch, merge the changes into other branches that have the same bug
• Warning: many people become over-excited about branching when they first start to use it
  • Keeping track of what's going on where can be a considerable management overhead
  • On a small project, very rare to need more than two active branches

5) Patching

• Often need a way to send or archive differences between two versions of a program
  • E.g., someone finds and fixes a bug in your open source software...
  • ...but doesn't have permission to commit the change to your version control repository
• Common to use the patch program to do this
  • Takes the output of diff and applies it to the original file to produce the modified file

6) A Better Way to Build

• Said when discussing Automated Builds that [Make] has turned into a clumsy programming language
• The same thing happens eventually to every other build management tool
  • So why not start with a real programming language, and embed a build management tool in that?
• [SCons] combines the most useful features of Make with the full power of Python
  • Instead of a Makefile, you write an SConstruct file
  • Use function calls in that file to tell SCons what you want to build, and how
    • Like Make, it has a rich set of default rules
  • But when you want to do something complicated, you can use any feature of Python you want
    • E.g., build up a list of filenames, fetch a build rule from a database, etc.

7) SCons Example

• Example: build either a normal or a debugging version of a program, and include an extra source file on Windows

# What does the program depend on?
dependencies = ['file1.c', 'file2.c']
if os.platform == 'win32':
    dependencies.append('win32.c')

# Which version are we building?
if 'debug' in COMMAND_LINE_TARGETS:
    Program('hello_dbg', dependencies)
else:
    Program('hello', dependencies)

• Pro:
  • Don't have to learn a new language
    • And the authors of the build tool don't have to create and maintain one, either
  • There's a debugger
• Con:
  • Much less widely used than [Make] or [Ant]

8) Persistence

• Often want to save the state of a running program
  • E.g., checkpoint a long-running program so that it can be restarted in case of a crash
• Don't want to have to rewrite this code every time the program changes
• Use a persistence framework like Python's pickle module
  • Walk through the objects in the program
  • Save the atomic values (integers, strings, etc.) as they are
  • Write lists, dictionaries, and other collections in a standard way
• Classes can define special methods to tell the framework what values to save

9) Pickling Example

Figure 27.2: Pickling

10) Object-Relational Mapping

• Increasingly popular alternative is to use an object/relational mapping framework like [SQLObject]
  • Define mapping between classes and database tables
  • Framework then creates code to translate objects to rows and back
  • Usually keeps a cache of recently-used objects to improve performance
  

  Figure 27.3: Object-Relational Mapping

• Pro:
  • Takes advantage of databases' strengths (high performance, concurrency control, etc.)
• Con:
  • Objects don't naturally fit into tables
    • E.g., how to represent many-to-many relationships?
  • Can be very hard to debug when things go wrong
    • I.e., when the generated code is doing what you said, but you asked it to do the wrong thing

11) Web Development Frameworks

• Handwritten CGI applications are going out of style
  • We've learned enough since the 1990s to write code at a higher level
• One widely used alternative is Java servlets
  • User writes a class whose methods do application-specific work
  • The servlet container loads the class, and calls those methods
• [Ruby on Rails] is another popular choice
  • [Ruby] is a scripting language with similarities to both Python and Perl
  • [Ruby on Rails] automatically handles object/relational mapping, constructs stubs of unit tests, etc.
• Python has several similar frameworks, such as [Django] and [TurboGears]
  • But none are as widely used or as well documented
  • Competition isn't always healthy...

12) Refactoring

• Refactoring is the process of cleaning up code
  • Often described in terms of "code smells" and corresponding cures
  • See [Fowler 1999] for a comprehensive catalog
• Very important to have unit tests in place before starting to refactor
  • Without this, you have no way of knowing what else your refactoring might have broken
  • [Feathers 2005] is an excellent guide to how to fit useful tests back onto inherited applications

13) Refactoring Examples

• Smell: method or function body runs to several pages
  • Cure: use Extract Method to break the method into smaller meaningful pieces
  • But do not break up arbitrarily, just to satisfy coding conventions
    • Each new method should make sense on its own
• Smell: function or method with many parameters
  • If a method takes eleven strings as parameters, sooner or later you'll pass them in the wrong order
  • Cure: store values as members of the object, rather than passing them as parameters
    • But it's bad to introduce members called param1 and param2
  • Alternative cure: introduce a new object that combines parameters into one value

14) More Refactoring Examples

• Smell: duplicated code
  • Cure: use Extract Method once again
  • But what if code is only almost duplicated?
  • Use Introduce Parameter to give callers a way to signal exactly what they want...
  • ...or use Pull Up Method to move shared code into parent...
  • ...and Form Template Method to have that shared code call something that each child class defines
• Smell: complex Boolean expressions in conditionals
  • Cure: Introduce Explaining Variable to give sub-expressions meaningful names
  • Particularly effective when it's used to turn nested if-then-elses into a lookup table

15) Refactoring Tools

• Refactoring complements design patterns
• Both provide a shorthand for communication, and make education easier
• Design patterns describe code at rest; refactorings describe code in motion
• But there's a difference: modern IDEs can do refactoring for you
  • Highlight a method, say "Rename", and the IDE finds and changes all the calls
    • Able to do this because it continuously re-parses source code as you type
  • Move methods up into parent class, split classes in two, and much, much more
  • If this doesn't convince you to upgrade from a dumb editor, I don't know what will...

16) Code Reviews

• Code reviews are more effective at finding bugs than testing [Fagan 1986]
  • A consequence of Boehm's Law: the earlier you find a problem, the cheaper it is to fix
• Unfortunately, very little has been written about how to read code
  • Which makes [Spinellis 2003] and [Spinellis 2006] all the more valuable
• Many open source projects require that changes be reviewed before being committed
  • See [Seamonkey] for example guidelines
• Not just for finding bugs: want to make sure that:
  • Team members are following style guidelines
  • They've implemented what they're supposed to
  • You understand how it works well enough to maintain it when they're gone
• Diminishing returns as programmers become more experienced, and more familiar with a project's idioms
  • But they're a great way for new team members to learn their way around
  • And an equally great way for experienced members to catch newcomers' mistakes early

17) Reading Code

• Print it out: paper has 4-10 times the resolution of even the best screen
  • And print out the ticket the work was done to resolve
• Sit somewhere comfortable
  • Away from email, the web, your pager, and other distractions
• Trace execution
  • Find an entry point (like main)
  • Skip over argument parsing, file I/O, etc.
  • Put a question mark beside everything that doesn't immediately make sense
  • Draw pictures of data structures, data flow, etc.
• Once you're done, go back and try to answer your questions
  • But don't cross them off: if you didn't understand it when you first read it, it needs to be clarified
• Finally, jot down overall comments

18) Code Review Checklist

• Is the code documented in a consistent and readable way?
• Are file, class, method, parameter, and variable names descriptive?
• Are all function inputs used? Are all required function outputs produced?
• Is the flow of control easy to follow? What about the class structure?
• Do conditionals cover all cases? Do all loops have an exit condition? Do they handle the zero-pass case?
• Do functions check that their inputs are valid?
• Do callers check return values for errors, or handle exceptions that might be thrown?
• Are errors handled in a standard manner? Are error messages descriptive and helpful?
• Are there any magic numbers or machine-specific filenames?

19) User Interface Design

• Doesn't matter what your software does if no one can use it
• Usability and user interface design are not black arts
  • Need a certain amount of natural talent to be a great UI designer
  • But anyone who can learn to program can learn to avoid basic mistakes
• Basic rules from [Johnson 2000]:
  • Focus on the users and their tasks, not the technology
  • Consider function first, presentation later
  • Conform to the users' view of the task
  • Don't make anything more complicated than it already is
  • Make it easy for users to learn new things
  • Deliver information, not just data
  • Design the interface to be responsive
  • Try it out on users, then fix it!

20) Paper Prototyping

• Paper prototyping is the fastest and cheapest way to design an interface
• Create rough sketches showing menus, buttons, etc.
  • Use Post-It notes for pulldowns
  • Keep it rough: the more polished it is, the less likely people are to give you critical feedback
• Find a volunteer, then play computer
  • Set them a task
  • Show them what happens when they click, type, etc.
  • Do not answer questions or provide hints, or get into a discussion of how to fix things
• Two or three sessions will be enough to tell you what you need to fix

21) Where To Go Next

• [Brand 1995] looks at how buildings can be designed to change and grow gracefully over time
  • Everything he says is directly applicable to large programs
• [Steele 1999] shows you how a great computer scientist thinks about a particularly hard problem
• [Margolis & Fisher 2002] describes a project at Carnegie-Mellon University aimed at making Computer Science a more congenial environment for women and other underrepresented groups
  • Note: while the gender ratio in computing as a whole is about six to one, the ratio in open source is closer to 200:1
• [Udell XXX] talks about how much more the web could do for scientists

22) The Rules

• A week of hard work can sometimes save you an hour of thought.
• Anything worth repeating is worth automating.
• Anything repeated in two or more places will eventually be wrong in at least one.
• The three chief virtues of a programmer are laziness, impatience, and hubris.
• It's not what you know, it's what you can.
• The deadline isn't when you're supposed to finish; the deadline is when it starts to be late.
• Never debug standing up.
• Tools are signposts, not destinations.
• Not everything worth doing is worth doing well.
• Code unto others as you would have others code unto you.
• Every complex file format eventually turns into a badly-designed programming language.
• Tools are amplifiers: they allow good programmers to be better, and bad ones to be worse.
• They call it computer science because it's experimental.
• Programs come and go; data is forever.
• There's no such thing as one program.
• Discipline matters more than genius; reality matters more than rulebooks.

23) Conclusion

• "A good teacher has a value above pearls, but a good student has a value above rubies."
• Thank you, and good luck