a life of coding

Java Static Class Object Fields and Weak References

noreply@blogger.com (ynniv) — Mon, 04 Aug 2008 20:01:00 +0000

In browsing the Mozilla Rhino code, I stumbled upon a garbage collection problem that is truly bizarre. Rhino uses thread local storage for the current Context object. TLS uses a weak dictionary, so it is important that there is no strong circular reference to the key. Years ago (CVS r 1.234), the rhino class Context used to have a static reference to the thread local storage object. This was a problem because even when an array is empty, it has a reference to the class of objects it can hold. Thus, the TLS key (the ThreadLocal object itself) is referenced by the Context class, which is referenced by the (empty) Context[].class, which is a value in the TLS map! This circular dependency caused rhino to leak memory whenever a thread that used JavaScript completed. The solution to this was to change the value to an Object[]. When there are Context objects in the array, the reference still exists, however once the array is empty, there is no longer a reference and the TLS weakref dictionary will cut that key/value pair loose.

The comment, original, and fix.

Wow, thats a lot of detail. Here's the summary: if you have static class fields, make sure they don't eventually point to the key of a weak dictionary. If find this unavoidable, you can break the loop by storing a value as an Object and casting any usage of it. You will still have to set this variable to null in order for GC to happen.

Firefox 3 is the Ugliest Browser Yet

noreply@blogger.com (ynniv) — Tue, 17 Jun 2008 20:00:00 +0000

It's a bit trite for a blog post, but seriously, Firefox 2 looked way better than 3 on Windows. If you customize the toolbar (via right click) to use "small icons", its marginally better, but people actually spent time working on the design of this UI, so I shouldn't have to fix it for them. Everyone would have been much better off with the third party Firefox 1.x theme Charamel.

I haven't used the Mac version yet, but it appears to have the same heinous icon problems. Look at Safari and try again please.

XPCOM nsISupports Proxy Crashes

noreply@blogger.com (ynniv) — Thu, 29 May 2008 21:39:00 +0000

In building an application based on XULRunner, we've uncovered a number of bugs in the Mozilla codebase. One that really bothers me is the implementation of asynchronous object proxies. First, here's a brief overview of nsISupports proxies.

XPCOM

XPCOM is a generalized system for calling methods on objects. These objects can be implemented in C++, JavaScript, or Python (using a 3rd party library called PyXPCOM). XPCOM is the mechanism for defining Firefox extensions and writing applications that use XULRunner.

Working With Threads

One handy feature of XPCOM is the ability to have your method call execute in another thread. One example of this is a background thread that wants to do something to the UI, such as update a progress bar. Similarly to Java, Cocoa, and some Windows programming, UI access in Mozilla has to happen on the main thread. Another common situation is sending commands to a worker thread from the main thread. In both cases, we have a simple command to issue, but a direct method call is the wrong way to accomplish this.

So XPCOM has a mechanism for defining proxy objects that look identical to the original object, but executes your request in another thread. There are two flavors of this, synchronous (PROXY_SYNC) which will block your thread, and asynchronous (PROXY_ASYNC) which queues a request in the other thread and continues on. One implication of asynchronous dispatch is that there will be no return value, since you don't wait for the method to finish.

The Async Warning

The page describing proxies contains a very specific warning: if you pass variables by reference and those variables are on the stack, bad things will happen. It turns out that this is a bit understated, and doesn't convey some of the subtleties, so I will try to clarify.

Asynchronous XPCOM Proxies Are Fundamentally Broken

Using them will likely cause you nothing but pain. This pain will be in the form of totally random crashes, with meaningless (but seemingly useful) stack traces. You will ask people for help, and they will have no idea where to start. Unless you are intimately familiar with the XPCOM message dispatch and QueryInterface source code, PROXY_ASYNC should be a keyword which in your mind expands to Bad Bad Evil Crashing.

Why? Well, it turns out that return values are implemented as "out" parameters, which is a fancy way of saying references on the stack, which you may have recently learned is something that you should never mix with PROXY_ASYNC. So, if you were to call any method on an asynchronous proxy that could return a value, it will corrupt memory. There's actually no uncertainty here, XPCOM will write over some random memory address which is probably on some poor thread's stack.

The Workaround

"Ah!" you say, I can use PROXY_ASYNC and still sleep at night as long as I only call methods that return "void". Well, you might sleep a night or two, but soon you'll realize that bad things are still happening. The reality is, there is no workaround (sorry!). Even if you avoid methods with "out" or "in out" parameters, and ones that return values, Mozilla will still punish you by calling exactly these sorts of methods internally. If you're using a scripting language (generally JavaScript), things are even worse for you, because it makes a bunch more of these calls to set up your script object.

The Documentation

This bug is currently documented in buzilla. Unfortunately, there seems to be no workable solution to the situation. On top of that, the people responsible for the code base don't know why anyone would even want to use asynchronous dispatch. So, that doesn't sound good. The only real fix is to find a way to remove any mention of PROXY_ASYNC from your code. Depending on your application, this may be an easy fix. So far for us, this has not been the case. (Mozilla was never intended to do what we're doing)

FYI: JavaScript is Lisp

noreply@blogger.com (ynniv) — Fri, 04 Jan 2008 17:44:00 +0000

Just a heads up in case anyone hasn't figured this out yet.

Some Things Never Change

noreply@blogger.com (ynniv) — Sat, 17 Nov 2007 22:05:00 +0000

I just finished watching a half hour video on computer graphics. It covers the basics of 2D and 3D graphics and basic animation in a way that the average person can understand, but with enough detail to provide basic understanding.

The information itself isn't particularly blog-worthy; there are plenty of resources for learning about computer graphics. There are two reasons that this is interesting to me. First, television seems to have a dwindling amount of educational content. I remember watching 3-2-1 Contact, Mr. Wizard, and NOVA as a kid. The programs currently on PBS don't seem to show the same interest in science, substituting fantasy, social interaction, and ironically, reading. This may be related to my perception that they are also oriented to a younger audience, but I can't help but feel that kids should be learning these things instead of watching TV, not by watching it.

Second, this video is from a program called For All Practical Purposes and was filmed in the late '80s. The computer systems used are from Symbolics, a company that made their own hardware and operating system written entirely in lisp. This idea is similar to the current Squeak project, which is written in smalltalk instead of lisp. The video is old, the computers are old, the technology is old, and yet, it isn't particularly different from the basic concepts that I learned at Georgia Tech a few years ago. The biggest changes have been in the hardware that allows us to animate using physical simulation, and have realtime rendering.

Modern computing doesn't seem that different from 80's computing, except that everything is faster, larger, less efficient, prettier, more connected, and cheaper (much, MUCH cheaper). Just today I learned of a London startup that is writing games that would have run on a 1.2 MHZ Atari 2600 in Macromedia Flash - which uses most of my 1,200 MHZ CPU. But ultimately, its all the same. In my eyes, this makes a stronger case for the idea of Computer Science, because if you ever really learned how computers work, you might understand them forever.

parent and parent and proto, Oh My!

noreply@blogger.com (ynniv) — Fri, 16 Nov 2007 20:40:00 +0000

Sometimes Javascript development can be rather scary. Unlike C or Java, which have well known standards, Javascript (like HTML) is often described by what works on one or more of the current implementations. On top of that, few people see Javascript as a legitimate language, and a vast number of novices are out there writing code that would make your skin crawl.

In short, if you are a knowledgeable programmer, quickly picking up Javascript is much harder than it should be. The syntax is C-like (generally regarded as a plus), the functions are first class (almost universally positive), and the object system is prototype based (at least smalltalk people like this). Soon there will be a common just-in-time compiler, and some people have suggested that it might displace currently popular languages. Personally, I wish that it ditched the prototype object system for a Python-like one, and had lisp-like macros, and just-in-time compilation can't come soon enough.

When a question about a Javascript feature like "x.__proto__" arises, it can be difficult to answer. In my experience so far, the best places to look are MDC and the IRC channel listed on the page.

What is __proto__?

When you make a Javascript "class" (ie constructing function), you can specify a prototype:

function Example() {
  this.foo = 1;
}

Example.prototype = {
  sample : function memberFunction() {},
}

This attribute is copied onto constructed objects as __proto__, like so:

ex = new Example();
if (ex.__proto__ == Example.prototype) print("same!");

__proto__ is a special attribute that is searched if an attribute is not found on the object, it is analogous to a parent class.

parent and __parent__ are substantially different. parent comes from the DOM, and is the tag that encloses the current tag. __parent__ is a special variable that is a reference to the scope that created the object. Like __proto__ is checked when a variable lookup fails. Where __proto__ is checked when the failure looks like this.missing_attribute, __parent__ is checked when the failure is in the form missing_global. According to Mozilla, __parent__ and __proto__ have been deprecated, which is unfortunate since they can be quite useful.

Debugging Python

noreply@blogger.com (ynniv) — Tue, 06 Nov 2007 04:41:00 +0000

Above all else, my greatest annoyance with python is the lack of good documentation and defaults. I bet that there is a group somewhere that knows everything there is to know about python... I beg of them, write that knowledge down!

Here's an example. I want to debug a faulty program:

x = [1,2,3]

def foo():
sum = 0
for i in range(4):
 sum += x[i]

import pdb
pdb.run("foo()")

I'm then prompted by pdb: (> replaced with ])

] [string](1)?()
(Pdb)

Good, now I'm in the debugger. To get things started, I type "c", return, and get:

Traceback (most recent call last):
File "[stdin]", line 1, in ?
File "c:\Python24\lib\pdb.py", line 996, in run
Pdb().run(statement, globals, locals)
File "c:\Python24\lib\bdb.py", line 366, in run
exec cmd in globals, locals
File "[string]", line 1, in ?
File "[stdin]", line 4, in foo
IndexError: list index out of range
]]]

You should notice that the last line is the standard python prompt (with the aforementioned character replacement to make blogger behave), not the debugger. I have finished my debugging session due to an error. Gee, it sure would have been nice to debug that, since my state is now lost! I was already using the debugger to run this code, why did it not catch this exception?

It turns out that exception handling is done by sys.excepthook, and pdb.run doesn't set the excepthook. Some searching turned up two options. The first is simple but crude - add the following to site-packages/sitecustomize.py:

Thomas Heller

import pdb, sys, traceback
def info(type, value, tb):
    traceback.print_exception(type, value, tb)
    pdb.pm()
sys.excepthook = info

The second is more sophisticated, and checks for interactive mode:

ActiveState Python Cookbook.

# code snippet, to be included in 'sitecustomize.py'
import sys

def info(type, value, tb):
   if hasattr(sys, 'ps1') or not sys.stderr.isatty():
      # we are in interactive mode or we don't have a tty-like
      # device, so we call the default hook
      sys.__excepthook__(type, value, tb)
   else:
      import traceback, pdb
      # we are NOT in interactive mode, print the exception...
      traceback.print_exception(type, value, tb)
      print
      # ...then start the debugger in post-mortem mode.
      pdb.pm()

sys.excepthook = info

Here's what I'm currently running:

# code snippet, to be included in 'sitecustomize.py'
import sys

def info(type, value, tb):
   if (#hasattr(sys, "ps1") or
       not sys.stderr.isatty() or 
       not sys.stdin.isatty()):
       # stdin or stderr is redirected, just do the normal thing
       original_hook(type, value, tb)
   else:
       # a terminal is attached and stderr is not redirected, debug 
       import traceback, pdb
       traceback.print_exception(type, value, tb)
       print
       pdb.pm()
       #traceback.print_stack()

original_hook = sys.excepthook
if sys.excepthook == sys.__excepthook__:
    # if someone already patched excepthook, let them win
    sys.excepthook = info

The original ActiveState script doesn't debug if you are running in interactive mode. To me, this makes no sense at all - thats a case where I specifically want to debug.

Alas, if the debugger shows an obvious error that could be fixed, python exceptions cannot be resumed. The code listed here will let you see the stack and move around in it, but your program is no longer running and can never finish where it left off. Allowing this requires call/cc, and as far as I can tell there is no plan to ever support that in python.

If this was helpful, or should be changed, let me know. I've just started using it myself.

Closures in Python

noreply@blogger.com (ynniv) — Sun, 12 Aug 2007 13:37:00 +0000

A closure is data attached to code (pretty simple, eh?). I use them for:

Replacing hard coded constants
Eleminating globals
Providing consistent function signatures
Implementing Object Orientation

(Isn't it funny that people rarely tell you what closures are good for?)
Here is a closure in python:

def makeInc(x):
  def inc(y):
     # x is "closed" in the definition of inc
     return y + x

 return inc

inc5 = makeInc(5)
inc10 = makeInc(10)

inc5 (5) # returns 10
inc10(5) # returns 15

Closures in python are created by function calls. Here, the call to makeInc creates a binding for x that is referenced inside the function inc. Each call to makeInc creates a new instance of this function, but each instance has a link to a different binding of x. The example shows the closure of x being used to eliminate either a global or a constant, depending on the nature of x.

import time
keepRunning = True
updates = []
def runLoop():
   while (keepRunning):
       for u in updates:
           u()

class foo:
   def __init__(self, x = 0):
       self.x = x

   def update():
       print self.x
       self.x += 1

f = foo()
g = foo(2)

updates.extend([f.update, g.update])

In python, all methods (but not functions) are closures ... sort of. The method definition foo.update closes the class foo. The value of g.update is a closure that stores the value of g and passes that as the first argument of foo.self, hence the first argument of a method in python is self. Details aside, it is important to note that the designers of python have gone out of their way so that you can pass g.update by itself to another function and have it continue to work correctly.

Caveats

In some languages, the variable bindings contained in a closure behave just like any other variables. Alas, in python they are read-only. This is similar to Java, and has the same solution: closing container objects. Closure of a dictionary or array won't let you assign a new dictionary or array, but will let you change the contents of the container. This is a common use pattern - every time you set a variable on self, you are changing the contents of a closed dictionary.

A Visualization of Visualizations

noreply@blogger.com (ynniv) — Fri, 03 Aug 2007 05:39:00 +0000

You really need to see this one first hand, so check it out.

Working with Python/C

noreply@blogger.com (ynniv) — Fri, 06 Jul 2007 23:13:00 +0000

Some very short notes for working with python from C. I'm using Python 2.4.x, YMMV.

Getting the current error state inside your debugger:

_PyThreadState_Current->curexc_type

Instantiating a "New Style" class (a subclass of "object"):

PyObject* args = PyTuple_New(0);
PyObject* dict = PyDict_New();

// works only with new style classes (subclasses of "object")
PyObject* result = PyObject_Call(classObject, args, dict);

Open Network Sockets on Mac OS X

noreply@blogger.com (ynniv) — Fri, 15 Jun 2007 16:54:00 +0000

This is one of those things that constantly annoys me. On Linux, netstat tells me the list of currently active network connections, including (often most importantly) listening connections. Just knowing that something is running on port 8080 tips me off that I probably have an Apache or Java EJB process running (or maybe a swiki) - a trip to localhost:8080 will answer my question. But what if this port isn't an HTTP server, and doesn't speak when connected to? Now you have a dilemma - there's no way of knowing what has this port open.

Fortunately this is a solved problem on Linux. netstat has some options that tell you the PID of the process with the port open. From here you can use ps to find the name of the process, its path, the user who started it, etc. BTW, you need to run netstat as root to see other people's PID's and those of services.

Well, you know what comes next - netstat on the Mac doesn't show PID's! WTF! Speaking of commands that Mac OS X doesn't have, fuser is missing as well. fuser on Linux tells you which processes have a specific file open - very useful if you're cleaning up files and one has is locked. Well, Mac OS X (BSD really) doesn't have fuser... but it does have a command called lsof. lsof isn't quite as user friendly as fuser. It only has one mode, which is to list every open file that is visible to you (this is a subtle hint that you should run it as root to see more files). This means that fuser <filename> roughly translates to lsof | grep <filename>. Very useful for finding that stray service that has outlived its welcome and is holding files hostage.

Still, the problem at hand is finding the PID of network sockets. It turns out that in POSIX, network sockets are pretty much the same as files. This means that they show up in lsof if you ask nicely. And since lsof shows PID's (it even gets fancy and shows the process name), it turns out to be the solution. So, here's the money shot:

sudo lsof -i -P

This produces something that looks like:


COMMAND    PID  USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
launchd      1  root    9u  IPv4 0x01d96e10      0t0    UDP *:137
launchd      1  root   10u  IPv4 0x020dbe8c      0t0    TCP *:139 (LISTEN)
launchd      1  root   11u  IPv4 0x020dbb38      0t0    TCP *:445 (LISTEN)
launchd      1  root   12u  IPv6 0x01d99c50      0t0    TCP *:22 (LISTEN)
launchd      1  root   13u  IPv4 0x020db7e4      0t0    TCP *:22 (LISTEN)
mDNSRespo   43  root    7u  IPv4 0x01d96ad0      0t0    UDP *:5353
mDNSRespo   43  root    8u  IPv6 0x01d96a00      0t0    UDP *:5353
mDNSRespo   43  root    9u  IPv4 0x03379d40      0t0    UDP *:5353
mDNSRespo   43  root   13u  IPv4 0x03379860      0t0    UDP 192.168.1.100:53891
mDNSRespo   43  root   14u  IPv4 0x032f82a4      0t0    TCP *:* (CLOSED)
mDNSRespo   43  root   15u  IPv4 0x02edb554      0t0    TCP 192.168.1.100:5000 (LISTEN)
mDNSRespo   43  root   16u  IPv4 0x020da098      0t0    TCP *:* (CLOSED)

... (lots more here)

If you don't have root access, you can still use lsof, but you won't see the plethora of system services and other users processes.

Python Techniques You've Been Googling For

noreply@blogger.com (ynniv) — Tue, 29 May 2007 16:43:00 +0000

I've been doing it, I bet you have too. Here's some topics that have come up lately, I'll be writing about each of them in time. Since you probably got here via Google, add a comment for the subject that got you here.

Closures
Debugging
Creating functions
Metaclasses
Performance Optimization
Objects in depth

asdf-install on openMCL

noreply@blogger.com (ynniv) — Sat, 17 Mar 2007 17:45:00 +0000

A Note: This has been sitting in my drafts folder for 10 months. I'm
posting it unedited because its better out then in, right?

---

This should be pretty trivial... after all, its included with the
latest openMCL, right? (I'm using the latest CVS)

Well, I've been fighting to get it working for almost a week, most
likely because I tend to ignore files named README. Having some
experience with CPAN and rubygems, it should really be possible to get
packages from a package repository without even knowing the language
that you are using. Why? Because newbies want to see results before
they have to invest time in learning new things. In the worst case,
an end user doesn't know, or even want to know, how to program at all.
Package repositories should really just work.

Here are my notes so far:

You need to compile asdf-install. To do so, you should be in the
directory containing asdf-install.asd . Then run:
(asdf:operate 'asdf:compile-op :asdf-install)
(asdf:operate 'asdf:load-op :asdf-install)
Now, make your asdf registry:

mkdir /usr/local/share/ccl/asdf-registry
cp asdf-

You need to create ~/openmcl-init.lisp with:
(require 'asdf)
(setf asdf:*central-registry*
'(*default-pathname-defaults*
#p"/usr/local/asdf-install/site-systems/" ; where
asdf-install puts things
#p"/usr/local/share/ccl/asdf-registry/" ; where i put
asdf-install.asd
))
(asdf:operate 'asdf:load-op :asdf-install)

;;; its really best to install GPG
(setf asdf-install:*VERIFY-GPG-SIGNATURES* nil)

The central registry setup is important - maybe
asdf-install/site-systems should be the same folder as the asdf
registry... I dunno. What I do know is that they both need to be in
the asdf search path (ie asdf:*central-registry*) or you won't get
very far.

I hope that I'm misguided, this is all a bad dream, and if I only
followed some magic instructions, this would be trivial. For now,
this works for me, and thats all that really matters.

Old And In The Way (stale fasl's)

noreply@blogger.com (ynniv) — Mon, 04 Dec 2006 02:37:00 +0000

SBCL 1.0 has been released, and as expected, it breaks some things on my Powerbook. One specific problem is stale "fasl" files. Honestly, I don't even know what they are... I suspect them to be some kind of optimized bytecode used by ASDF to improve loading speed, but really (as long as they work) I don't care what they are.

SBCL will stutter something like this:
debugger invoked on a SB-FASL::INVALID-FASL-VERSION: #<SB-SYS:FD-STREAM for "file /usr/local/lib/sbcl/site/rt-20040621/rt.fasl" {11A024F9}> is in native code fasl file format version 70, but this version of SBCL uses format version 71.
Searching cliki revealed that these are intermediate files that can simply be deleted. They suggest that deleting files by hand is tedious and have a nice code snippit that should make things magically refresh in the future... I'm a little more pragmatic:
find /usr/local/lib/sbcl -name '*.fasl' -exec rm {} \;
Works fine afterwards.

Update: I was a little quick to declare victory. After executing this, i was unable to (require 'asdf), since the fasl for asdf had been deleted. I reinstalled (executed sbcl-1.0/install.sh) and the problem is fixed. This seems to invalidate my previous suggestion... Maybe I should have filtered the search to only include files older than a certain date. The cliki code snippit is probably the better way to go for now.

Python Developers Wanted: Dead or Alive

noreply@blogger.com (ynniv) — Wed, 22 Nov 2006 04:13:00 +0000

I've been working at a new gig in Boulder, Colorado. We're writing an earth rendering engine, a la Google Earth. Why bother? Unlike the competition, we're going to be putting lots of our imagery on hard drives so that you can take it off the net: in your car, on a plane, in your backpack. You can sync changes when you get home, or use a cellular connection for low bandwidth data like local searching and geolocating your friends. If you've wanted it, hopefully we're on it (but feel free to send suggestions or leave comments).

I think 3d earth rendering is going to be big (maybe I've just read Snow Crash too many times), and recent announcements by Microsoft seem to confirm my suspicion. This blog doesn't get many comments, but usage statistics betray the silent masses who frequent it looking for KML tools. Are you tired of writing web services and pounding KML into new Google Earth features? Does writing a car navigation system 12 hours a day in python using a real 3d world API (we won't poke too much fun) sounds like your cup of tea? Maybe you're tired of Carmack taking credit for your OpenGL code. I heard somewhere that start-ups aren't en vogue anymore, but I know there's other people out there who regret missing the bubble. If you're confident in your abilities and can write the code to prove it, we're looking for you.

It would be nice if you lived in the US and wanted to move to Colorado (its sunny and warm even when the slopes are open and there's plenty of outdoors to go around. Send us an email.

How Python is Killing Lisp

noreply@blogger.com (ynniv) — Sun, 19 Nov 2006 15:11:00 +0000

These are tumultuous times for lisp. Once the bastion of language academics and hackers and shunned by those of industry, it's received much lip service from successful dot commers, most notably Paul Graham. Lisp has been around a long, long time (its the second oldest language still in use, behind fortran), and its gotten very mature in that time. Industry languages continue to steal features from lisp, and some modern languages like python and ruby are more lisp than C. Still, lisp does not have the popularity one might expect. What is it that's holding it back while python prospers?

Lets start with what it gets right. It is said that lisp is very expressive, allowing extremely complicated ideas to be expressed succinctly (and hopefully quickly) by those fluent in the language. That's a rather heady idea, especially considering it is being applied to a practical industry. What this means in practice is that a great programmer is less limited by lisp than C++ or Java. This is really important because there are no natural limits on the productivity of a programmer. Unlike artisans and construction workers, there are no physical processes that limit how fast he can accomplish a task. The best person of another trade might be hundreds of times better than the novice, but they are limited by nature. Take away that limitation, and a handful of the best people could accomplish amazing things.

What makes lisp expressive aren't complex features like Object Orientation, but powerful simple ones like closures. The simplicity of these features let you build new complex ones as times change. Aspect Oriented Programming was part of the original Object Oriented lisp system based on closures. Continuations are a great way to deal with the non-linear flow of web applications.

Many of these powerful features have been adopted by new languages: Python can generate closures and has list comprehensions. Still, they don't contain all of lisp's power, and yet a search for open-source lisp applications on sourceforge returns 345 results to python's 6,571. Where are the hordes of elite programmers looking to change the world? Its easier to write things in lisp, right?

What lisp is getting wrong, opening the door for other lisp-like languages, is the new user experience. This means documentation, discoverability, sample code, libraries, and books. Its forums with solutions and smiling people. Unfortunately the lisp community is confident that their horse will win in the end, and they're just not willing to put in the effort. Other languages whose success is not so certain are more motivated to please. Python is one guy's foolish attempt to change the world, and you know that he spent a lot of time working to that end.

What lisp needs is is dedicated people behind it, laying the foundation of what people want. Specifically, people who are not currently lisp users. Given a python executable, some syntax, and knowledge of "import", "dir", and "help", anyone can learn python. Lisp should also be easy for newbies; there is even less syntax! SLIME is a great package, but please, stop hurting the cause... people who don't use Emacs are terrified by it.

Even the first step of a newbie has too many decisions: picking a lisp interpreter. There are simply too many of them, and their features are often inconsistent and incomplete. Perl, python, and ruby all managed to get one interpreter running on many platforms fairly uniformly. I don't worry about the networking or threading model inconsistencies between python on my mac and python on my linux server.

They also all have package managers that just work. This is a critical feature that just needs to work. CPAN really set the bar for package mangers in general. Asdf-install is okay, but it can be inconsistent across runtimes and packages don't always install correctly.

There are simple things as well. I am constantly irritated by the lack of prompt features in lisp interpreters. I'm lucky if they even recognize control characters... command history is a blessing. I haven't seen one that has any manner of auto-complete (something based on list-all-packages and do-symbols would be fine, thank you).

Its the unfortunate nature of lispers to work on the really long term things which are totally useless to their contemporaries. While I really appreciate that Paul Graham has decided to work on his own version of the language, I suspect that his lisp oriented incubator company would benefit more from a simple, usable IDE for sbcl and openmcl.

Ultimately, I am hopeful. Lisp is something that I believe in, and I'll keep trying to do more with it. If any language is ready to transcend old habits, this one is. Most things have already been tried in it once (distributed computing, hardware implementation, graphical editing), the next big thing is probably already in there somewhere waiting to be harnessed.

Back to Basics: Attending UC Berkeley From Your Lap

noreply@blogger.com (ynniv) — Thu, 01 Jun 2006 03:40:00 +0000

Background

You're three years into high school, and bored out of your mind. You live in, have always lived in, have never left, central Connecticut. Its a far cry from the mid-west or the deep south, but you still find it difficult to learn more than they print in your text books, or what 9 out of 10 people know. An hour due north east and you might have been the subject of a half-indie blockbuster movie... but you aren't nearly that lucky. You're looking for an out, but you aren't even old enough to drive a car.

As it turns out, you're a lucky bastard. If you haven't yet guessed, this was my youth - years of frustration in trying to pursue something that no one understood. The benefit? Being able to gripe about how you pulled yourself out of nowhere by your own bootstraps. My verdict? Griping gets you no where, just do a better job than I did. After all, you're a lucky bastard. The reason that you are a lucky bastard - that you have no excuses - that this situation simply no longer exists - is simple: the Internet.

The Appetizer

If you are an athlete, painter, musician, or gear head, the Internet is probably quite useful. Sites like Wikipedia provide you with information that you would have never had access to (personally, I would have killed at least one or two people for that kind of resource). They'll help you sharpen your technique and learn the latest trends, even if you live in a cave (one with net access). But these areas require physical skills, and probably coaching. You must practice them for years, especially if they are competitive. They are intrinsically tied to the people available to you. On top of that, the best information for you is probably not available on the net at all - the best athletes and painters probably don't write web pages.

Ah, but maybe you're not. Maybe you are a computer geek, as I am. Spending more time trying to describe yourself in terms of bits and algorithms than figuring out who you are in the first place (btw: not healthy). If this is the case, then you sir (or hopefully, madam: 'cause computing need chicks) are in luck. While you might live in the sticks of Alaska, Peru, or Iowa, you are on the forefront of the computer industry. Don't lament not being in the Valley (crowded), don't make excuses of Route 128 (cold). The secret that expensive real estate doesn't want you to know is simple: the computer industry elite is everywhere are nowhere, wherever there is 'net. Does high school seem more interesting yet?

The Entree

So, young padawan - where to start. Some rules of thumb:

Don't use commercial products
Don't worry that your work doesn't look professional
Don't listen to what most people say - they don't know anything anyway. Especially if you are in High School.
Don't ignore everyone - some people know more than you (or they) think they do.
Don't get involved in politics or religion too much.
For the love of God, don't write C++. (You'll thank me later)

Read a lot. Paul Graham, Golden Age SciFi, Cyberpunk, the classics. I regret not reading more.
Exercise regularly. I fought what my parents forced on me, but you picked a mostly sendentary career and the human body requires exercise. Learn it early, and you'll appreciate it forever.
Learn Java. It has its problems, but its your best bet. Download src.jar and learn how everything works.
Finish things. When you're teaching yourself, its easy to leave things half done. Make a list of things you are working on and when they will be finished.
Use open source tools. Use Linux. Download source code for things and see how they work.
Talk to people. Just as a storyteller needs an audience, you need users. Programming is not about computers, its about solving people's problems.
Contribute to open source projects. You are contributing to the wealth of the world, and you can put that on your resume. Plus, you don't have to interview for them (suits bad!).
Think about what you are doing. Part of what you are doing is learning, so pay attention. Honestly, you don't want to succeed too much too early, or the allure of skipping college will be too great.
Oh yeah, go to college. You won't want to, and you might not enjoy it, but if you want to succeed (and who doesn't?), you have to. Its hard to explain why, so you'll just have to trust me. If you find it too easy, there will be a world of seriously cutting edge projects looking for volunteers. If you find it too hard...

Finally, the title of this post - attend UC Berkeley. Its easy! Really, anyone can do it - I do it at work. Plus, its free - how can you say no to that?

In a bid to put other schools out of business, the top programs are putting scores of class resources online - including podcast audio of lectures. I have been listening to CS 61A with Brian Harvey at work. Being a little underwhelmed with my GeorgiaTech undergrad education, I was curious to see what (if anything) I missed out on. So far, I like the clarity and confidence that Harvey has - but then, I thought the same of Kurt Eiselt. The course seems significantly faster and denser than my Tech experience. I suppose a program like UCB can expect more of its undergrads.

If you are interested in learning Lisp (or anything for that matter) or testing the breadth of your skills, I recommend checking out the plethora of courseware available. If you're trapped in High School computer class, step up and ask your teacher to incorporate intro level college courses. The first couple months of a class like CS 61A will teach you more than a year of AP Computer Science. And I cannot stress enough that all of this is free. If you have the time, you are losing nothing by checking it out.

Just Desserts

Convinced that you are alone in your thirst for knowledge? You could not be more wrong. Get together with your friends and work on things together. Networking is the name of the game in software. If you are in the Atlanta area, check out atlHack.org. We're like a support group for caffeine addicts and computer programmers. But it isn't hard to start your own - websites and time are both cheap now. If you do, drop us a line and we can start a network of hackers.

Whatever you do, make sure that you're in it for the right reasons. If you enjoy what you do, you'll get more out of it and do a better job. No startup has been founded by people who don't enjoy what they do. Plus, I don't like working with people who don't enjoy what they do, so do me a favor.

Airport ICAO SQL

noreply@blogger.com (ynniv) — Mon, 22 May 2006 04:55:00 +0000

Getting simple information on flights is a real pain in the ass. There are airport codes (IATA and ICAO), airplane codes, and airline codes (IATA and ICAO). Here's the good news:

There is an existing database for IATA and ICAO airport codes.
There aren't too many airplane codes, and they aren't as important
IATA airline codes seem to be out of fashion, and will be officially replaced by ICAO codes soon.

I've scraped Wikipedia to put together ICAO airline codes in SQL. I won't be maintaining this list, but if you just want to cover the basics, you're welcome.

Game Lispers

noreply@blogger.com (ynniv) — Tue, 09 May 2006 04:34:00 +0000

I've made some pretty technical blog posts so far. They've covered Objective-C, Java, Lisp, KML / GoogleEarth, python, ruby on rails, Mono, C++.... lots of topics that I don't generally talk about IRL because if I did, I wouldn't have any friends ( atlHack aside). At the same time, I don't get many (well, any) comments to any posts, so for a long time I was pretty certain that no one was listening to the blog either. Well, it turns out that I was wrong, and I get a lot more blog hits than I ever suspected. Some of them are via search engines, so in addition to page hits I have the search queries that brought you guys here.

My last post / rant focused mainly on some of the queries that I see, so I won't indulge myself with yet another rant. Instead, I'd like to talk about one query that seems to show up every month for a least a couple of hits: "naughty dog lisp". Of every statistic on the awstats page (and there are a lot of them) this is my favorite. Every time I see the count increment I smile, knowing that my blog is getting hit by some of the most elite hackers in the world. Why do I say that? Lets break this down:

Game hackers:

Are excited about their work. Any time people are excited about their work, they will be naturally better at it than if it bores them.

Deal with a lot of hard problems: real-time performance optimization, media (sound, graphics, physics), and concurrency.
Generally push the hardware to its absolute limits, just for the fun of it.

Lisp hackers:

Are looking for ways to make themselves more efficient.
Aren't scared of complex systems, but probably appreciate simplicity and elegance
Are working on the bleeding edge of development, even if that edge happens to be 50 years old. Lisp does not have hundreds of Dummies books.
Value function over form. There is no flashy IDE for it. It doesn't generate documentation from your comments. Its value is not in decoration, but in raw substance.

Given that people searching for "Naughty Dog Lisp" are in both of these groups, I would say they are a formidable group. But enough about them, I have real substance for this post.

I have started experimenting with OpenMCL's OpenGL bindings. For a long time, I have battled with graphics programming. The majority opinion is to use C and OpenGL, or C++ and a scenegraph library (Ogre3d is nice, and cross-platform). I appreciate the performance of these solutions, but I have always felt limited once my programs exceeded a few hundred lines. There isn't much you can do in a few hundred lines of code, especially if you've got visions of Oni in your head. So I have played with a number of programmer-efficient alternatives, like python, and my own 1/2 of common lisp, Commotion. (Commotion has somewhat transformed into a graphical programming language, akin to Max/MSP, which might be best implemented on top of lisp - but more on that later.) But PG's talks are eating away at the back of my mind - its time to leave the cheap substitutes behind and take on the real deal.

Which brings us to the listener interactive part of this post. I know that there are people out there reading this, and some of them are chasing the same unicorns that I am. If this is you, I want to hear about it. There's a comment button down there that's getting lonely. Anonymous comments are allowed, so there are no account-related excuses. Have you written a game in lisp? Screensaver? Music? Maybe you're just interested, or your Intro to Computing class is in scheme and you want to do something useful with it. Anything goes, except for comment spam, which recieves swift death.

A Little Break From Coding

noreply@blogger.com (ynniv) — Tue, 04 Apr 2006 06:43:00 +0000

AWStats has brought to my attension some incidents that I find a little disturbing... Let me present to you the current search queries for ynniv.com in April:

Term	Hits
recursive drawing java	2
naughty svn	1
rails .htaccess rewrite	1
itunes 5	1
dependency injection frameworks	1
recursive drawing	1
moving to another country	1
coding autorun	1

First off... "naughty svn"... 3rd page, baby (due to a mishap link to another blog post, "Naughty Dog Software"). ... Why would someone search for this? Is your versioning system being a bad little boy? Does it need discipline? Is it wearing black panties or something? People, it versions your text files - leave it alone! Nothing good will come of molesting svn, its only there to serve your basic versioning needs!

Next up, "itunes 5". Apparently, a blog of mine made it to page 13 of "itunes 5" on google. Which begs the quesiton, What were you doing on page 13?! After the first 10 or so pages, would you not reconsider your search term? And more importantly, why am I even on page 13?!? The only thing I had to say was that iTunes 5 looked a lot like iTunes 4 with the graphical contrast cranked up. I'm not sure if my google ranking says more about Google's PageRank system, or people's opinions of iTunes 5.

Now, my favorite: "moving to another country". I made page 4 on this one. I like this for a number of reasons. First, I think more people should move to another country. Not in the "this is my country, get a move on" sense, but the "everyone cool is going somewhere else" sense. Why? Because I think that the United States is simply too cocky. We're pretty far behind on a lot of new things (the painfully obvious, glaring example is Broadband 'net access), and yet, we think we're the bomb dug. For instance, were you aware that the United States is Ferrari's largest market?

Plain and simple, we have too much money and no idea what to do with it. So, if you haven't yet, and are interested in, I highly recommend spending some time in another country. No, don't go on vacation - thats just moving your US lifestyle to Ibiza for the week. The French call this "le bullshite". If you have the opportunity, study abroad. Trust me, you have no idea how important this is until you've done it.

Go to a place that has a high cost to quality-of-life ratio, and you'll understand how real people live. In places like Costa Rica and Peru, you could retire on the cost of that Ferrari - and live a good life too. Guess what - there's even Internet access. Think about that - with $150k in the bank, you could buy your own castle in Costa Rica, and still read Slashdot. Now, you should be warned that you'll need to own and know how to use a shotgun. But, how hard do you really have to work to bank $150k for retirement? I met a guy from California who did just this.

Questions? Comments? ynniv-web@this-url.

Blending KML

noreply@blogger.com (ynniv) — Tue, 21 Mar 2006 04:56:00 +0000

Update: We're hiring python programmers to write a car navigation system in our earth rendering engine (a la Google Earth).

Google is the best thing that's happened to Keyhole. They've transformed a cool visual toy into a platform for GIS information, and the key to their success is twofold. First is the open KML (XML) file format - anyone can make simple content. Second, Google Earth is free for limited use. The combination is magic - developers are eager to create content, consumers are eager to view this content, and Google gets rapid desktop application adoption and the ability to roll out future advertising.

The KML format is easy enough to read and write by hand for small amounts of meaningful data, like placemarkers and pointers to raster files for ground overlays. What's currently lacking is a free way to convert lots and lots of meaningless data, like 3d models. There are a few commercial tools out there, most notably SketchUp (recently purchased by Google, 8 hour trial available), but nothing free for the casual hacker... until now.

kml_export.py is a python script for blender that supports meshes and diffuse lighting. Not much, I know, but KML doesn't support much 3d. This is really a quick-release, rough cut, leaves much to be desired release - but it gets the job done.

Installation:

Put kml_export.py in your Blender scripts directory
(windows = "c:\Program Files\Blender Foundation\Blender\.blender\scripts",
mac = "/Applications/Blender/blender.app/Contents/MacOS/.blender/scripts",
linux = "~/.blender/scripts"?)
Edit kml_export.py to adjust the placement constants (lat, lon, altitude, scale)
If you're using Windows, you'll need to download PyXML. Unarchive the PyXML folder and move the "xml" subdirectory to the same folder as kml_export.py.

Your scene needs to contain at least one mesh object to produce useful output.

Deploying App that use dylibs on Mac OS X

noreply@blogger.com (ynniv) — Mon, 13 Feb 2006 00:31:00 +0000

dylibs can save you space, but only if you already have the library
installed. When shippings apps to Mac users, this can be a problem -
having to do anything other than drag an icon to your disk (or
decompress a .app.tgz) cuts into user appreciation. One way around
this is to include everything a user needs into the applicaiton
bundle. For frameworks this is cake: XCode provides a "copy
frameworks" build phase, and any frameworks that you add to this will
be copied into the bundle before they're linked (if not, make sure the
phase happens before the library link phase).

But I wouldn't be writing this post if that were the end of it.
dylibs are dynamic libraries, like DLL's on Windows, .so's on unix,
and the binary part of a Framework. dylibs contain references to
their locaiton on disk, which is a problem if you plan on moving them
to package them in your app's bundle. Mac OS X includes two utilities
that will help you fix this: otool, and install_name_tool. XCode's
"run script" build phase provides a convenient place to patch the
dylibs after you've copied them. Here's a short script that I use to
fix the paths of libgauche for my application:


#!/bin/bash
EXECFILE=${BUILT_PRODUCTS_DIR}/${EXECUTABLE_PATH}
LIBPATH=${BUILT_PRODUCTS_DIR}/${SHARED_SUPPORT_FOLDER_PATH}
NEWLIBPATH="@executable_path/../SharedSupport"

# space separated list of libraries
TARGETS="libgauche.dylib"
for TARGET in ${TARGETS} ; do
LIBFILE=${LIBPATH}/${TARGET}
TARGETID=`otool -DX ${LIBPATH}/$TARGET`
NEWTARGETID=${NEWLIBPATH}/${TARGET}
install_name_tool -id ${NEWTARGETID} ${LIBFILE}
install_name_tool -change ${TARGETID} ${NEWTARGETID} ${EXECFILE}
done

Troubling SVN Error

noreply@blogger.com (ynniv) — Fri, 16 Dec 2005 15:25:00 +0000

There is a somewhat rare subversion error out there, in which you can import files but not add them. This is rather infuriating, as you often add your whole project to source control, work on it for a month, and then try to add a file and find that subversion has just broke. Really, it didn't just break, it was always broken and you never knew it. Googling for the answer won't help (well, maybe it will now), because no one seems to know the answer. But, lucky you! I have your answer!

Here's the situation: you are running a website package (like drupal). This package puts up fancy 404 error pages, possibly sends you to the root page. When you type in http://www.mywebsite.com/this_page_doesnt_exist.html, instead of an ugly "404 Document Not Found", you get a pretty "302 Found http://www.mywebsite.com/404.html", which causes your browser to redirect. This is good for users - people don't get much out of the traditional error message. This is bad for svn - when you say "svn add" and then commit, it says "Hi! This file already exists!", which causes svn to tell you: "svn propfind request failed", "302 Found"... less than useful.

So, you already see the answer now, cunning user: go to your apache subversion config, and add the simple statement: "ErrorDocument 404 default" (and reload apache). Users get pretty errors, subversion gets less confused, you get to add files. Server defenestration averted.

Thats it! You're done! Go and add new files with abandon!

Lisp in Games: Naughty Dog's Jax and Daxter

noreply@blogger.com (ynniv) — Mon, 05 Dec 2005 01:52:00 +0000

Here is a brief clipping from a Gamasutra article that talks about Lisp used for writing console games. The really really short version: lisp allows for awesomely simple cooperative threading (via continuations) and a short development cycle (using interactive mode on the console), if you can find a language that runs on your console, and programmers who know lisp.

----

GOAL rules! Practically all of the run-time code (approximately half a million lines of source code) was written in GOAL (Game Object Assembly Lisp), Naughty Dog's own internally developed language, which was based on the Lisp programming language. Before you dismiss us as crazy, consider the many advantages of having a custom compiler.

Lisp has a very consistent, small set of syntactic rules involving the construction and evaluation of lists. Lists that represent code are executed by evaluating the items that are in the list; if the head of the list is a function (or some other action), you could think of the other items in the list as being the parameters to that function. This simplicity of the Lisp syntax makes it trivial to create powerful macros that would be difficult or impossible to implement using C++.

Writing macros, however, is not enough justification for writing a compiler; there were features we felt we couldn't achieve without a custom compiler. GOAL code, for example, can be executed at a listener prompt while the game is running. Not only can numbers be viewed and tweaked, code itself can be compiled and downloaded without interrupting or restarting the game. This allowed the rapid tuning and debugging, since the effects of modifying functions and data structures could be viewed instantaneously.

We wanted creatures to use nonpreemptive cooperative multi-tasking, a fancy way of saying that we wanted a creature to be able to execute code for a while, then "suspend" and allow other code to execute. The advantage of implementing the multi-tasking scheme using our own language was that suspend instructions could be inserted within a creature's code, and state could be automatically preserved around the suspend. Consider the following small snippet of GOAL code:

(dotimes (ii (num-frames idle))
   (set! frame-num ii)
   (suspend)
      )

This code has been simplified to make a point, so pretend that it uses a counter called ii to loop over the number of frames in an animation called idle. Each time through the loop the animation frame is set to the value of ii, and the code is suspended. Note that the value of ii (as well as any other local variables) is automatically preserved across the suspend. In practice, the preceding code would have been encapsulated into a macro such as:

(play-anim idle
   ;; Put code executed for each time..
   ;; through the loop here.
   )

There are other major compiler advantages: a unified set of assembly op-codes consistent across all five processors of the Playstation 2, register coloring when writing assembly code, and the ability to intermix assembly instructions seamlessly with higher-level code. Outer loops could be written as "slower" higher-level code, while inner loops could be optimized assembly.

GOAL sucks! While it's true that GOAL gave us many advantages, GOAL caused us a lot of grief. A single programmer (who could easily be one of the top ten Lisp programmers in the world) wrote GOAL. While he called his Lisp techniques and programming practices "revolutionary," others referred to them as "code encryption," since only he could understand them. Because of this, all of the support, bug fixes, feature enhancements, and optimizations had to come from one person, creating quite a bottleneck. Also, it took over a year to develop the compiler, during which time the other programmers had to make do with missing features, odd quirks, and numerous bugs.

Eventually GOAL became much more robust, but even now C++ has some advantages over GOAL, such as destructors, better constructors, and the ease of declaring inline methods.
A major difficulty was that we worked in such isolation from the rest of the world. We gave up third-party development tools such as profilers and debuggers, and we gave up existing libraries, including code previously developed internally. Compared to the thousands of programmers with many years of C++ experience, there are relatively few programmers with Lisp experience, and no programmers (outside of Naughty Dog) with GOAL experience, making hiring more difficult.

GOAL's ability both to execute code at the listener and to replace existing code in the game at run time introduced the problem of memory usage, and more specifically, garbage collection. As new code was compiled, older code (and other memory used by the compiler) was orphaned, eventually causing the PC to run low on free memory. A slow garbage collection process would automatically occur when available memory became sufficiently low, and the compiler would be unresponsive until the process had completed, sometimes taking as long as 15 minutes.

Wired's Top 10 Computer Malfunctions

noreply@blogger.com (ynniv) — Tue, 08 Nov 2005 18:30:00 +0000

http://wired.com/news/technology/bugs/0,2924,69355,00.html?tw=wn_tophead_1

Breakdown:

2 rocket controllers
2 radiation therapy controllers
1 act of cold war sabotage that blew up a Soviet gas line
1 hardware (Pentium floating point error)
3 network exploits
1 telephone network software

Lessons of the Day:
if you're writing embedded software for rockets or medical devices, make sure you're right.
the network is never secure.
don't trust the software you stole from the CIA.