July 24, 2008
Redmonk
Web Green: The Open Web Foundation
Today at O’Reilly’s OSCON, David Recordon announced the formation of a new foundation, the Open Web Foundation, with the goal of fostering development of open standards and providing an incubator for working out IPR and patent issues before standards are too widely deployed.
The Open Web Foundation is an attempt to create a home for community-driven specifications. Following the open source model similar to the Apache Software Foundation, the foundation is aimed at building a lightweight framework to help communities deal with the legal requirements necessary to create successful and widely adopted specification.
This is an awesome step and one that has been necessary for a long time. The Apache Foundation has provided a great model for software incubation, but to date there has not been an appropriate place to work out issues with new specifications. The Open Web Foundation will provide that place.
The OWF has a mailing list up for anyone wanting to get involved or follow the discussion. Details on foundation membership, etc, are still forthcoming but knowing the participants I’m certain things will be aboveboard.
I’m really excited to see the OWF come together - the environment in which web technologies are developed has changed, and the OWF is there to help provide a buffer for these new specs to be developed out in the open. As I wrote in Web Green: Cultivating The Open Web:
Like sediment in a river, or potting soil in a greenhouse, each layer we put down supports and affects the ecosystem that grows out of it. We take IP, ethernet, and their like completely for granted - they’ve been standardized and implemented across a worldwide network. That layer is foundation and fertilizer for the next: HTTP, SSL, HTML, XML, and the feed variants that have become the everyday building blocks of our applications and services. These are now settling into the foundation for the services we’re building now: near-real-time publishing and social software stacks. These, in turn, will provide for what comes after, and the philosophical foundations we build into this layer will profoundly affect the health of the next.
So here’s to the OWF: May the foundations of the next web be as open and implementable as the foundations of the first web were, and more.
grass image from pygment.com for whose proprietor I cannot find contact information.
4 Tips for a Successful Man-venture
After a fun trip skydiving with my father and two brothers this past week, I wrote up a guest post on Dustin’s new blog on 4 Tips For A Successful Man-venture.
Though a tad awkward to say, ["Man-venture"] sums up easily our two-day combination of fun, adventure, companionship, and mutual support. Ours was a roaring success, and we all came away from it both tired and invigorated, and swearing to do it again in 6-12 months.
July 22, 2008
Aaron Swartz
The Percentage Fallacy
There's one bit of irrationality that seems like it ought to be in behavioral economics introduction but mysteriously isn't. For lack of a better term, let's call it the percentage fallacy. The idea is simple:
One day I find I need a blender. I see a particularly nice one at the store for $40, so I purchase it and head home. But on the way home, I see the exact same blender on sale at a different store for $20. Now I feel ripped off, so I drive back to the first store, return the blender, drive back to the second store, and buy it for $20.
The next day I find I need a laptop. I see a particularly nice one at the store for $2500, so I purchase it and head home. But on the way home, I see the exact same laptop for $2480. "Pff, well, it's only $20," I say, and continue home with the original laptop.
I'm sure all of you have done something similar -- maybe the issue wasn't having to return something, but spending more time looking for a cheaper model, or fiddling with coupons and rebates, or buying something of inferior quality. But the basic point is consistent: we'll do things to save 50% that we'd never do to save 1%.
At first this almost seems rational -- of course we're going to do more to save more money! But you aren't saving more money. With both the blender and the laptop, you have the chance to save $20. Either way, you're going to have another twenty in your pocket, which you can spend on exactly the same things later on. Yet we behave differently depending on whether we got that twenty by skimping on a small purchase or skimping on a big one. Rationally, if driving back to the store isn't worth $20 when you're buying a laptop, it isn't worth $20 when you're buying a blender.
On the other hand, don't those small savings tend to add up after a while? If you start blowing $20 every time you buy a trinket, you're soon going to be out of disposable income. Meanwhile, spending several thousand dollars is much rarer, so isn't it OK to slack off a bit on such occasions?
If we work to save 50% on everything, big or small, that's the equivalent of saving 50% of our money altogether. Whereas if we only try to save fixed amounts on every purchase, how much we save is dependent on how many things we buy.
So which is the real irrationality? I'm not entirely sure of the answer.
July 18, 2008
Morbus Iff
Wikipedia and the Zodiac Killer
Zodiac, a serial killer whose identity still remains unknown, continues to make headlines, most recently in the murder of Megan Touma, when a newspaper received a letter from someone purporting to be the killer, who will "start using my role-model's signature". A friend on IRC noticed that the John F. Kennedy page on Wikipedia had been defaced with a message, purportedly from Zodiac. Within 30 minutes, the message had been decrypted:
vandaljzing wikipedia is so fun it is funner than killing people or having sex because when you kill people you only destroy their bodys but when you vandalize this website you kill the soul of the poor queer pig slaves that devote their lives to the embarkment that i am now destroying with my own bare hands
Rough timeline of events:
- Google indexed the same message in the William McKinley Wikipedia article.
- sbp noticed that there were 26 unique symbols, suggesting a simple substitution cypher.
- sbp provided a simple symbol-to-character hash, for use in online solvers.
- ianivs used a cracker to solve it in 6395041 attempts.
- The folks in #wikimedia found the template changes.
July 15, 2008
Morbus Iff
Drupal Tough Love #2: Signatures for Forums 5.x-2.3
Over at Drupal Tough Love, chx and I just reviewed Signatures for Forums 5.x-2.3 which "provides user signatures that will be familiar to users of popular forum software" such as "the administrator can choose the input filter for signatures", conditional signatures that are hidden "if a post is under a particular length", and showing the signature only once per conversation.
July 07, 2008
The Mocking Eye
Moviesneak — Stretch your movie ticket dollar
Ever sneak from one movie to another at the movie theatre, after only paying for one ticket? Well, I have. And I like it.
This can often be done by looking at the showtimes ahead of time and finding movies that are times relatively near each other. Although this is sometimes thwarted by placing the temporally adjacent movies in different sections of the theatre, there are often opportunities for this type of sneaking.
But instead of bothering to study movie showtime tables and such, I’ve written a little python script to do it for me. It’s rather rudimentary. Here’s how to use it:
- Get moviesneak.py
- Get showtimes for a theatre in your area.
- The format moviesneak accepts is Google Showtimes, which can be accessed by googling
showtimes <zipcode>, or using this url:http://www.google.com/movies?near=<zipcode>. - Then select all the showtimes in a particular movie theatre, starting from the review stars, not including the theatre’s address.
- Copy these into a file somewhere.
- The format moviesneak accepts is Google Showtimes, which can be accessed by googling
- Run moviesneak:
moviesneak.py showtimes-file [optional time threshold in seconds].- By default, the threshold is 15 minutes (900 seconds). The threshold is how long before or after the end of one movie the following one should be. So a 15 minutes threshold means the following movie can start 15 mins before the current one ends, or 15 after (remember that even though it starts 15 mins before the current ends, you won’t have to leave early, you’ll only miss the previews)
The code is under a BSD license, so just about anything can be done with it. If you have time to muck with BeautifulSoup or just plain regex, it would be nice if moviesneak could query google showtimes (or any other source) directly. The algorithm is very simple, and pretty ugly, but it can be plugged into pretty surroundings easily (such as a web interface).
Edit: I haven’t bothered writing a chaining algorithm (i.e. find the longest sequence of contiguous movies) mostly because I really can’t watch more than 2 movies in a row at a movie theatre. However, one shouldn’t be hard to write, since the corpus size is small enough even horrible algorithms will chug through quickly. Maybe when I’m bored enough/free enough I’ll write one to extend moviesneak.
Share ThisJuly 02, 2008
John Cowan
Dorian pictures
See the <a href="http://www.flickr.com/photos/7731152@N03/tags/dorian/">tagged Flickr photostream</a>. I will add to this.</content>
June 30, 2008
Aaron Swartz
Capital and its Complements: A Summary
The following is a non-technical summary of Brad DeLong's May 2008 paper Capital and Its Complements.
Adam Smith explained that in all countries with "security of property and tolerable administration of justice" citizens would spend all their money (capital), either on consumption or investment, causing the country's economy to grow. After some contention, later economic studies tended to bare this out: a shortage of capital wasn't always the bottleneck, but when it was, removing it could lead to extraordinarily rapid growth.
The problem for poor countries is that, because of high mortality rates (which require more children to have some survive) and low educational levels (which mean those children can find productive employment quickly), they have high population growth and thus low capital-to-labor ratios. Worse, trade allows you to spend your money buying manufactured goods from overseas, for which you have only your very cheap labor to provide in return. The result is that it requires an enormous amount of domestic investment to improve capital-to-labor ratios.
And so rich country economists made "the neoliberal bet" on behalf of poor countries: they hoped that loosening restrictions on international capital flows would send capital rushing in to poor countries and build their economies, the same way that Great Britain's massive investment in a young United States (in 1913 Britain's foreign assets equaled 60% of its domestic capital stock) built up that country.
But what ended up happening was exactly the opposite. Yes, NAFTA led US companies to invest the $20 to $30 billion a year on manufacturing in Mexico that its boosters predicted, but that investment was more than outweighed by the $30 to $40 billion a year fleeing the country from Mexico's wealthy wanting to invest it in the United States. Why? In part because the US was more politically stable, and thus a safer investment climate. And in part because the US treats its own workers so poorly -- with productivity rising 35% since 2000 while real wages remain flat -- it provides an excellent investment opportunity.
But meanwhile, all this investment in the US was dwarfed by the Chinese acquisition of our debt (and thus the political risk it represents). China needed to do this, since US purchase of their exports is the only thing funding the manufacturing-led industrialization of a massive portion of their economy; there would be massive dislocation if that funding dried up.
"Recognition of these facts came slowly." First, Larry Summers said it was our unsustainable current account deficit. (That was the 1990s; today that deficit is four times as large.) Later, economists thought it must have been our large budget deficits. Then they began thinking it was the run-up in housing prices. But that, it is now clear to most economists, was the result of a bubble. And yet the flow of capital to the US continues. But, perhaps even more frighteningly, it could stop at any moment.
Morbus Iff
Help save the International Cryptozoology Museum
Friend and well-known cryptozoologist Loren Coleman writes that his International Cryptozoology Museum is in dire need of financial support, else it might be closed down due to an IRS audit:
"The [IRS's] audit has gone to five or more levels of meetings and appeals, as I have attempted to explain what being a “cryptozoologist” is all about, that I do make my living this way, and that the museum is an essential part of the overall plan ... To the IRS, the museum verges on being a hobby ... and it needs more income (even if donations) to support itself ... The museum has to make money, or it ceases to exist ... I need to get donations to pay the mortgage for the house-museum where it is now, to keep it from foreclosure, and to pay for the other bills associated with the museum."
I've known about the museum for years and, altruistically, supported it in any way I can. Unbeknownst to me, I was one of the first two to contribute long ago on May 14th, 2005. Back then, Loren had some books donated that reeked of cigarette smoke and, due to allergies, needed to find them a better home. I offered to take them off his hands but, more importantly, to give him some cash to hunt up replacement copies for the museum.
This weekend, I offered my continued support:
"In the last 24 hours, the first “Museum Principal Donor / Life Member” contributed $1050.00. This individual “Morbus” challenges others to join him in donating at the $1000 level to [meet] the goal of keeping this institution open."
Please, do what you can to help the museum. The alternative is that this publicly available resource, in a field already entrenched with a lack of answers, will disappear. Even those who aren't involved or interested in cryptozoology should appreciate the need to keep this information easily available and approachable.
June 28, 2008
Christopher Schmidt
Technocentric Thinking
Chad writes:
I know their “motto” is “Don’t Be Evil” .. but I think it should be “Don’t Be Smart” instead.. this is some dumb thinking from Google. Trust me.. I know better than Google on how I want to download and install my software.
This is just the latest in a whole lot of similar statements I’ve seen from many people across the web in a variety of situations talking about how “I know how to manage my machine”, with the underlying meaning being something like “You should act as if people who are working with your software know how to work their computers.”
When I put it that way, does it really sound right? Is there anyone who thinks that the *majority* of users of Google Earth actually know how to run their machines? Is there anyone who thinks that it makes sense for Google to build and QA two different install mechanisms — one for technical users who know what they’re doing, and one for those who don’t?
Very few companies the size of Google do anything on a whim. I expect that some thought went into the development of the Google Earth downloader. The fact that the thinking is not centered around technically competent users is just evidence that Google doesn’t need to target the early adopters; it’s not a sign that what they are doing is ‘Bad’ or ‘Stupid’.
Technical users are few and far between in the mass market. Google Earth is targeted towards the mass market. Just like all software that is targeted towards a mass market, there is nothing ’stupid’ about removing tools that the majority of users don’t need or care about: By doing so, you limit the number of people who are going to end up confused by your tool, and that’s not a bad thing when you care about the majority of people instead of a technical elite.
Printing Maps; Twin Mountain DRG
So, I’m going camping this week. While camping, I plan to be hiking the White Mountains relatively extensively, from my home base at Lafayette Campground in Franconia Notch.
One of the best part of planning hiking trips is maps. I love maps. I love looking at them, finding things, exploring things. I love having them handy to reference, I love browsing around them.
However, when you’re on a trail, looking at maps on a computer doesn’t work particularly well. As a result, I’m trying to print maps, and finding that it’s a somewhat difficult proposition.
I have DRGs from the libremaps project: These are great. I’m looking at o44071b5, o44071b6, and o44071a6, all downloaded from the Libre Map Project. Unfortunately, I then went to go open c6, only to find that… it’s not available. Argh! I know that it exists: I can check the features that should be on it in the Terraserver DRG WMS, but no Twin Mountain quad.
Oh well. Not a huge deal, just annoying; I’ve got other maps that cover the area. Now the problem is that I can’t figure out how to take a tiff like these and print them over multiple pages on a home printer. Anyone got any tips or tricks on that? Printing on one page is somewhat useful, but not nearly as much as I’d like; I also don’t want to spend all day cropping and fighting with tools. If I have to do that, I’ll just survive with what I’ve got.
I’m going to go out to the local map store and see if they have a nice Atlas for the Whites; if so, I’ll take it. It’s worth it to have a little bit of extra knowledge about a place — and who can’t use more maps?
June 26, 2008
Redmonk
Web “Green”: Cultivating The Open Web
It’s been a while since I’ve posted about what’s going on in the DiSo community, and I had started to prepare a list of recent developments to share, but on the way I felt that there was a theme I wanted to address first.
The DiSo Project is first and foremost about enabling/creating a new category of social-networking-enabled websites, not restricted to the large silos but grown organically at the edges of the web - the small and independent sites that are the forerunners and foundations of the communities we now enjoy. How can we best provide a fertile environment, one that encourages, protects, and nurtures this growth?
grass image from pygment.com for whose proprietor I cannot find contact information.
Fertile Foundations
One theme that’s been cropping up on the conference circuit lately, thanks to Chris Messina, Dave Recordon, Jeremy Keith, and others, is this idea of “building the open web“. The internet (based on public, open technical standards), and the early www (based on public, open hypertext formats and protocol specifications), gave “the web” we know its heart and soul. How did that happen, and what will perpetuate the process?
Like sediment in a river, or potting soil in a greenhouse, each layer we put down supports and affects the ecosystem that grows out of it. We take IP, ethernet, and their like completely for granted - they’ve been standardized and implemented across a worldwide network. That layer is foundation and fertilizer for the next: HTTP, SSL, HTML, XML, and the feed variants that have become the everyday building blocks of our applications and services. These are now settling into the foundation for the services we’re building now: near-real-time publishing and social software stacks. These, in turn, will provide for what comes after, and the philosophical foundations we build into this layer will profoundly affect the health of the next.
Building the Open Web
So for the next ecosystem of social and community applications to thrive, we need to make sure that these aspects - public, freely-implementable formats and open standards - are a part of the web as we know it now. Thankfully, it’s happening - witness the growth of open, enabling technoliges like:
- Microformats, basic specs for marking up machine-readable data in human-readable web pages (XFN, hCard, hCalendar, hAtom, hEtc)
- OpenID, open identity solution for web services
- OAuth, an HTTP-based protocol for authentication between services
- XRDS-Simple, which provides discovery for various web services and makes inter-app cooperation that much easier.
- XMPP, a real-time, distributed messaging system that can be integrated into other services.
All these are publicly developed and freely implementable, and active communities have evolved around them to discuss, implement, and evangelize them. This is what building the open web is about: collaborating to build a web that is larger than any company or organization - a web that will encourage new growth.
New Growth
So, given all the effort we’re putting into creating a web that is fertile ground for what’s coming next… what’s coming next? Here’s a look at a few areas DiSo is focusing on as we work on the building blocks of the distributed social network:
Identity
OpenID has focused a lot of attention on putting the User’s online identity back under their control. Rather than maintaining an account on each and every site they use, the User can maintain one or more OpenID accounts, using them as credentials on any of the 10,000+ sites that accept an OpenID for registration and login. With the technology in place, we turn our attention to what identity means, how much of that identity we’re willing to share, and with whom.
Activity
Since the mid 90’s we’ve been working on the problem of how to track what our friends and contacts are doing online, and figure out where the stuff that’s really interesting and relevant to us is happening. Look at the social network silos, and you’ll see that a huge part of what they offer users is the ability (or at least impression) that the user can know what their friends are up to. Sites like Twitter and FriendFeed are making progress on bringing this activity tracking into the light, but to really distribute it all there’s still a lot of work to do.
Here at the edges, we’re making it easy to agregate your own activity, and working on ways to track/follow updates of your friends activity in near-real-time.
Messaging
With OpenID providing a common form of identity, we’ve begun looking at what services can be enabled using that endpoint. One of the services we’re exploring is distributed messaging - friend requests, subscription requests, and direct messages - directed to that endpoint, authenticated via OAuth, and filtered by a messaging service based on user preferences.
Cultivating the Open Web
As the builders - or growers - of this web, it’s our responsibility to look beyond the IPO, beyond the ad-sell, beyond the current crop of buzzwords. We must decide that we’re going to invest in, and give back to, the ecosystem that has supported us. Think of it as Web “Green” - protecting and nurturing and stewarding the web ecology.
June 25, 2008
Christopher Schmidt
DjangoGraphviz: Visualizing Django Models
Earlier today, a coworker was trying to work out a diagram for the models in the Django app that I’ve been working on internally, to visualize the relationships between them. I did a quick Google, and found a reference to DjangoGraphviz, a super-handy little chunk of code.
DjangoGraphviz did exactly what I needed it to, with no problems at all. (My only complaint is that it requires the DJANGO_SETTINGS_MODULE to be defined in order to get the –help output, somewhat unintuitive.) The software quickly generated a .dot file which I was able to turn into a lovely PDF, and print. I’ve now got a copy on the desk of each of the developers using the codebase, and I think it’ll prove a lovely piece of reference.
So, if you want a quick visualization of your Django models, and you can install graphviz, I highly recommend DjangoGraphviz to do it.
(Note that the wiki page itself recommends a couple other things more ‘built in’ to Django, which are new to me: I didn’t try these, I just stuck with DjangoGraphviz, which did what I wanted.)
June 23, 2008
The Mocking Eye
Teaching
Share ThisThe primary task of a useful teacher is to teach his students to recognize ‘inconvenient’ facts - I mean facts that are inconvenient for their party opinions. And for every party opinion there are facts that are extremely inconvenient, for my own opinion no less than for others. I believe the teacher accomplishes more than a mere intellectual task if he compels his audience to accustom itself to the existence of such facts. I would be so immodest as even to apply the expression ‘moral achievement’, though perhaps that may sound too grandiose for something that should go without saying.
June 21, 2008
Morbus Iff
Code shui, NeoVictorian Computing, and Beauty
Friend Mark Bernstein promotes "software as craft" with the phrase NeoVictorian Computing. Jeremy recalls that "Part of his argument is that software creators have something to learn from the ideals of the arts and crafts movement: the software world is full of soulless bits and bytes, and maybe we would all be a little happier if we embraced handcraft ... During the talk, I remember Bernstein proposed that software creators should sign their work as a painter signs a painting, which is a lovely visual metaphor that I hope to keep around." And Greg Wilson has a book called Beautiful Code.
Happily, I already agree - they're all echoes of my own belief in "code shui", be it XML (a Morbus Rant from 2002 on "why beauty is important in computer file formats") or in code from 2004 ("His style is quite unique. [Morbus' AmphetaDesk] source reads almost like a paper, instead of terse code. He documents his code well and I've thus far found nothing that was very hard to understand. Best of all, its so un-Perl. He doesn't seem to use really clever tricks to do simple things, so the code has been very easy to understand").
June 20, 2008
The Mocking Eye
An svnwiki in Python
On a whim, I wrote a little wiki type thing in Python that sits on top of an SVN repo. It’s incredibly basic, and basically lets you browse a repo and edit files. Natively, it supports markdown as its default display mechanism, but it would be trivial to teach it the meaning of file extensions and have other view templates.
The intended use was for a personal notebook type thing, which I decided to abandon. Basically, it’s a lot like Jottit, except you actually have all your data, and can replicate it between locations. Yes, it also sounds like git-wiki,but I only found out about that after I’ve finished coding this version.
Although I decided not to use it, someone else my find it useful, at least the codebase. I am offering it here with absolutely no warranty, and you can use it however you like, you can attribute or not, whatever. Since this was built for private use, i.e. no public access, I was going to integrate grep into it, and other such utils, leveraging unix text processing for search, mass editing, etc. So one may be interested in continuing that. I did not implement a facility for adding pages easily, though that’s a trivial piece of coding.
Code: svnwiki.tar.gz Change the base variable to reflect the location of your repo.
Requirements:
- web.py
- pysvn (on debian this is the
python-svnpackage rather thanpython-subversion) - markdown
June 19, 2008
Kevin Reid
Apple's Sampler file format, and SBCL SB-SPROF report generation
Apple's Sampler is a profiler based on the principle of periodically collecting the entire call stack of the executing threads, then summarizing these stacks to show what occurs frequently; primarily, as a tree, rooted at the bottom of the stack, where each node shows the number of times that call sequence was found on the stack.
SBCL's sb-sprof is a profiler which also collects call stacks, but its summary report is much less useful to me as it does not provide the per-branch counting; just top-of-stack frequencies and a caller/callee graph.
Therefore, I examined Sampler's file format and wrote code to generate it from sb-sprof's record.
The file is mixed text/binary, LF line endings. The grammar, as far as I've determined it, is:
"@supersamplerV1.0" LF
"@symboltableV1.1" LF
(TAB int32<id> TAB int32<unknown>
TAB text<symbol>
TAB text<library-path> TAB text<library-path> LF)*
"@end" LF
(
"@threadV1.0" TAB int16Hex<thread-id> LF
(
TAB int32<1> int32<0> int32<1> int32<count of stack-frame> (int32<stack-frame>)* LF
)*
)*
"@end" LF
where by "int32" I mean a big-endian 32-bit (unsigned?) integer (i.e. four not-necessarily-ASCII bytes), and by "int16Hex" I mean a 16-bit integer in hexadecimal (i.e. four ASCII bytes).
"id" is an arbitrary identifier for this symbol. "unknown" is occasionally nonzero, but I don't know what it means. "symbol" is the name of a function/method found on the stack. "library-path" is the pathname to the object file it was loaded from (relative in the case of a standard framework, e.g. "Carbon.framework/HIToolbox.framework/HIToolbox").
"thread-id" is an identifier for the thread, which should occur as an "id" in the symbol table; the upper 16 bits evidently must be 0. Thread symbol table entries have a name and library path which is the string ("Thread_" int16<thread-id>); I have not confirmed whether this is necessary.
Each entry in a @thread block is one sampling of the whole stack of that thread. I do not know what the 1, 0, and 1 mean, but the fourth integer is the number of frames on the stack; immediately after are that many integers, each of which is an id from the symbol table.
Files generated from this structure are accepted by Sampler, but not always by Shark; I don't know why, and my attempt at tracking it down made it seem to depend on the size of the trace file.
Here is code to generate such a file from sb-sprof data; it should be loaded in the SB-SPROF package:
(defun write-32 (index stream)
(write-byte (ldb (byte 8 24) index) stream)
(write-byte (ldb (byte 8 16) index) stream)
(write-byte (ldb (byte 8 8) index) stream)
(write-byte (ldb (byte 8 0) index) stream))
(defun write-sampler-data (stream)
(format stream "~&@supersamplerV1.0~%")
(format stream "~&@symboltableV1.1~%")
(let ((symbols (make-hash-table :test 'equal))
(buffer))
(loop with gen = #x00002121
for i below (- (samples-index *samples*) 2) by 2
for debug-info = (aref (samples-vector *samples*) i)
do (unless (or (member debug-info '(trace-start))
(gethash debug-info symbols))
(let ((index (incf gen)))
(setf (gethash debug-info symbols) index)
(write-char #\Tab stream)
(write-32 index stream)
(write-char #\Tab stream)
(write-32 0 stream)
(let* ((*print-right-margin* nil)
(*print-miser-width* nil)
(name (princ-to-string
(if debug-info
(node-name (make-node debug-info))
(format nil "Thread"))))
(real-info
(when (typep debug-info 'sb-di::compiled-debug-fun)
(sb-di::compiled-debug-fun-debug-info debug-info)))
(source
(when real-info
(sb-c::debug-info-source real-info)))
(pathname
(when (and source
(eql (sb-c::debug-source-from source) :file))
(sb-c::debug-source-name source))))
(nsubstitute #\/ #\Newline name)
(format stream "~C~A~C~A~C~A~%"
#\Tab name
#\Tab pathname
#\Tab pathname)))))
(format stream "~&@threadV1.0~C~4,'0X~%" #\Tab (gethash nil symbols))
(flet ((out ()
(when buffer
(write-char #\Tab stream)
(write-32 1 stream)
(write-32 0 stream)
(write-32 1 stream)
(write-32 (length buffer) stream)
(dolist (sym buffer)
(write-32 sym stream))
(terpri stream)
(setf buffer nil))))
(loop for i downfrom (- (samples-index *samples*) 2) to 0 by 2
for debug-info = (aref (samples-vector *samples*) i)
do (cond
((eql debug-info 'trace-start)
(out))
(t
(push (or (gethash debug-info symbols)
(error "No symbol assigned to ~S" debug-info))
buffer))))
(out)))
(format stream "~&@end~%"))
Usage:
(with-open-file (s "sampler.trace" :direction :output
:element-type :default
:if-exists :supersede)
(sb-sprof::write-sampler-data s))
This code generates a noninteractive Sampler-style tree report from SB-SPROF data.
(defun make-bucket ()
(list 0 (make-hash-table :test 'equal)))
(defun put (bucket name)
(let ((h (second bucket)))
(or (gethash name h)
(setf (gethash name h) (make-bucket)))))
(defun build-tree ()
(loop with root = (make-bucket)
with current = (list root)
for i downfrom (- (samples-index *samples*) 2) to 0 by 2
for debug-info = (aref (samples-vector *samples*) i)
do (cond
((eql debug-info 'trace-start)
(dolist (bucket current)
(incf (first bucket)))
(setf current (list root)))
(t
(let ((name (node-name (make-node debug-info)) #|kludge|#))
(push (put (first current) name) current))))
finally (return root)))
(defun twiddle-tree (bucket)
(let (entries)
(maphash (lambda (k v) (push (list* :i k (twiddle-tree v)) entries))
(second bucket))
(list (first bucket) entries)))
(Usage for printing a report: (twiddle-tree (build-tree)).)
Aaron Swartz
Last Goodbyes
It's minutes to midnight and I'm hurriedly packing. Early tomorrow morning I catch a flight to Boston and start my new life. I haven't really gotten much of a chance to pack until now, because I've spent the past few days in a rush of meetings, getting in my last goodbyes for everyone I know in San Francisco.
It's been great seeing everyone, but like most locals, they're all puzzled as to why I'm leaving. I've been struggling to explain why. When I say the weather, everyone just laughs. When I say San Francisco is too loud, they start arguing. When I say it's the people, they tell me to find a better group of friends.
And the thing is, they're right. It's none of these. I've been spectacularly unable to articulate it, but the real answer is simpler and more prosaic. And now, after great thought and struggle, I realize the answer is simply this: Cambridge is the only place that's ever felt like home. It's that simple. And when you put it that way, it's clear why I have to go.
So goodbye Stanford, goodbye Palo Alto; goodbye south bay, goodbye peninsula; goodbye Change Congress, goodbye Creative Commons; goodbye Mission, goodbye SOMA; goodbye friends, goodbye loved ones; goodbye San Francisco, home to everyone I've ever loved. You'll always have my heart.
Scenes
"God, I'm so sick of this stuff. Can't we just go home?" she wines. "Jesus," I say, "would it kill you to go one more place?" It's been a long hot day in strange, busy New York City, and we're not exactly at our best. In fact, the combination of heat and exhaustion has turned our love bitter, brought on the darkness and recriminations. Its at moments like these, the dark depths of a relationship, that you wonder how things could ever work. As we walk down the steps we hear a subway car approach. We accelerate, running to catch it. Its doors open. We're moving faster now, pushing our way through the bustle of Manhattanites to make it. The bell sounds and I jump inside and hear the doors whoosh closed behind me. I spin around only to see her trapped on the other side of the glass. I put my hand up to it, but the train accelerates and she's left standing there, just another face in the crowd.
"Hey, want to see the game? Want a ticket to the Giants game?" I do not, in fact, want to see the game -- this or any other game. I hate sports. Yet the scalpers, apparently unaware of this, insist on trying to sell me one. That's what I get for walking near the ballpark, I guess. As I curse my choice of scenery, a cop pulls up. He lowers his window and leans out toward the scalper. The scalper hands him a ticket and the cop speeds off. "But he didn't pay!" a man in a suit walking by complains. "Cops get a special deal," explain the scalper. The man in the suit laughs and marvels at the scene.
It's weird being back at Stanford in the summer. Everything's so empty, nobody's around. Well, not nobody -- there seems to be some action near the main quad. There are drum kits spread around and golf carts and purple uniforms lying about. But most of all, there are people -- a bunch of students just standing around awkwardly. I'm about to ask one of them what's going on when a bell rings and a voice shouts "Background!" Suddenly all the students snap to attention, begin walking in perfect lines with bookbags slung over their shoulder, bicycles ridden in perfect formation. These aren't students at all, I realize with a lurch -- they're extras. It's disconcerting. A police guard is at the side, keeping kids from running over the camera crew. I ask her what they're filming. "Disney's High School Musical," she says quickly, trying to keep a student from cycling over the director's cart.
John Cowan
Dorian Sion Cowan
<p>My grandson Dorian was born at 9:08 PM yesterday, June 17, 2008 (New York time). He weighed 9 lb 0.9 oz (4110 g) at birth, and was 22 inches (56 cm) long. And he is the Best Baby In The World.</p> <p>(Well, when I say that, I make a mental reservation in favor of Irene, Dorian's mommy, who is now almost 21 but was certainly the Best Baby in her day.)</p> <p>Baby and mother are doing wonderfully well -- Dorian is starting to breastfeed very nicely, and already knows a great many Proto-Indo-European roots. Irene's Caesarean incision is still very sore, and the IV is in her hand, not her arm, which makes handling him a little awkward for her. Her best friends have been hovering around the two of them, and so have Gale and I as far as we have been able. They will be coming home Friday morning.</p> <p>Anyhow, I sang him a lullaby the night he was born, not that he needed it -- he was pretty well drifting off anyhow. But even though my voice was cracking, I needed to sing it to him. It's by <a href="http://en.wikipedia.org/wiki/Fred_Small">Fred Small</a>, and is called <a href="http://lyricsplayground.com/alpha/songs/e/everythingpossible.shtml">"Everything Possible"</a>. This is the slightly altered version of the chorus that Dorian actually got: <p>You can be anybody you want to be,<br /> You can love whomever you will.<br /> You can travel any country where your heart leads,<br /> And know I will love you still.<br /> You can live by yourself, you can gather friends around,<br /> Or find one special one,<br /> And the only measure of your words and your deeds<br /> Is the love you leave behind you when you're gone.</p> <p>And this is the second song he heard from me, this morning when I stopped by to see him:</p> <p>Rockabye Dorian, on the tree-top<br /> When you are fed, your poop will go plop<br /> When you have plopped, your diaper we'll change<br /> And then you'll be cleaned up and happy again.</p> <p>Okay, it doesn't quite rhyme, but it's <i>his</i>.</p> <p><i>Dorian, if you are reading this, you already know your grandfather is a crazy old man who embarrasses the hell out of people. You'll live this one down too.</i></p></content>
June 17, 2008
Aaron Swartz
Moving On
In November 2006, I moved to San Francisco because I had to: my company got acquired and us moving out was a condition of the agreement. It was the first time I'd ever actually lived in San Francisco, as opposed to just visiting, and I quickly realized that although it was a fun place to visit, I couldn't stand living here.
Even after all this time, I can't really put my finger on what it is I don't like -- in fact, I suspect it's probably harder for me now to explain it than it was when I first came here. The first thing that comes to mind is how loud the city is. I want a place where I can live quietly and focus on my work; but San Francisco is filled with distractions. There are always crews tearing up the street, trains that are delayed, buses that have broken down, homeless people begging, friends having parties, and so on. It's impossible to concentrate and without my concentration, I feel less like me.
The other big problem is that San Francisco is fairly shallow. When I go to coffee shops or restaurants I can't avoid people talking about load balancers or databases. The conversations are boring and obsessed with technical trivia, or worse, business antics. I don't see people reading books -- even at the library, all the people are in line for the computer terminals or the DVD rack -- and people at parties seem uninterested in intellectual conversation.
And so I'm moving back to Cambridge, Massachusetts -- Harvard Square in particular, the one place I've ever been to that brings a special delight to my eyes, that warms my heart just to see. Surrounded by Harvard and MIT and Tufts and BC and BU and on and on it's a city of thinking and of books, of quiet contemplation and peaceful concentration. And it has actual weather, with real snow and seasons and everything, not this time-stands-still sun that San Francisco insists upon.
I miss Boston; I'm excited to go back.
But I'm also sad to leave my responsibilities in San Francisco. One of which I'd particularly like your help with. I've been honored and overjoyed to help Lawrence Lessig get his Change Congress project off the ground. If you haven't heard, he's trying to build a national movement to get the corruption out of Congress; to pass public financing of public elections, earmark reform, and other pressing concerns.
But they need a full-time day-to-day tech organizer. Someone who knows how to blog and who the bloggers are and can keep them in touch with the community. Someone who knows enough about technology to know the tools that can be built and should be. And someone with enough drive and talent to make sure those things get built. It's a dreamy job and I hope there's someone out there who will take it from me. A more formal write-up is on the Change Congress blog.
Thanks for everything.
June 16, 2008
Christopher Schmidt
GeoJSON 1.0 Release
From the GeoJSON mailing list:
The GeoJSON Authors are proud to announce the finalization of the GeoJSON 1.0 Specification.
Representing more than a year’s worth of community discussion and development, the GeoJSON specification describes an easy to use, extensible format for transferring geographic data over the web. With support in more than 20 different applications, GeoJSON is already quickly becoming a de facto standard for transferring geographic data in a JSON format. The finalization of the spec represents the final step in formalizing the GeoJSON format for encoding this data.
More information on GeoJSON can be found at http://geojson.org/ , or from the GeoJSON mailing list at http://lists.geojson.org/listinfo.cgi/geojson-geojson.org .
June 13, 2008
The Mocking Eye
DIYBio
Yesterday I partook in the 2nd ever meeting of the diybio group in Cambridge, MA.
The meeting was hosted at BetaHouse this time rather than the less experiment friendly Asgard “irish” pub. I got there early, but soon enough other diybionauts — there is no consensus yet around nomenclature, though biohacker seems to hold most sway — began to filter in. We started by chatting and discussing the proposed activity: the good old party trick of extracting DNA from various things (I posted a very similar protocol in this blog before).
The demographics seems skewed pretty much exclusively towards tech professionals and young academics in their 20’s. The majority don’t have degrees in biology, but obviously, everyone is very much interested in the subject or they wouldn’t be involved. A small transhumanist contingent was also to be found.
When everyone had arrived, we proceeded with the experiment. We used oatmeal, apples, and humans as sources for DNA samples. I was one of the brave few who volunteered their precious genetic data, and I daresay our DNA ended up vastly superior. Either way, it was lots of fun, and in mere minutes we had actually done some biology in a kitchen. Modest beginning foretelling future greatness, perhaps? After we were finished we quickly covered the biological processes involved in our experiment, and sat down to considering the future of the organization.
DIYBio aims to be a beacon of responsible and safe amateur involvement in biotech. A major part of its mission is to provide education and guidance on techniques and procedures. But while pursuing these lofty, and slightly nebulous goals, we will do lots of fun biological stuff, too!
When a $100 Transformation was suggested (i.e. modifying an organism for $100) I replied that it may come closer to $200 in the end, One Transformation Per Child someone quipped in response. A lofty goal, but a worthy one! Another idea floated was to field an iGEM team not backed by an academic institution.
A sort of “minutes” of the meeting can be found on this thread of the diybio google group.
The picture on the left? That’s me with the test tube of human DNA. The DNA is the milky white substance visible between the two layers of liquid in the tube. There is also some DNA attached to the toothpick I am intently examining 
Don’t phage me, bro!
Share ThisAaron Swartz
Is Undercover Over?
My latest piece for Extra! is now up:
Is Undercover Over?: Disguise seen as deceit by timid journalists
It's about the rise and fall of undercover journalism. Here's an excerpt:
Undercover reporting has a storied history. Nellie Bly, famous for traveling around the world in 80 days, also did a famed investigation of the conditions in insane asylums for the New York World. Bly feigned insanity for a series of physicians before being committed to a lunatic asylum. There she documented rotten and spoiled food, freezing living conditions, frigid bathwater, abusive nurses and relatively sane fellow residents. "What, excepting torture, would produce insanity quicker than this treatment?" she wondered. The series, later published as the book Ten Days in a Mad-House, created a sensation, and Bly was asked to join a government investigation of asylum conditions.
June 09, 2008
Aaron Swartz
How to Promote Startups
When people talk about how government can promote startups, there seems to be a fairly standard consensus: we need more economic inequality. Lower income and capital gains taxes provide more incentive to work, looser labor laws make it easier to fire non-performers, and large private wealth funds provide investment capital.
But having been through a startup myself, I think there's much more you can do in the other direction: decreasing economic inequality. People love starting companies. You get to be your own boss, work on something you love, do something new and exciting, and get lots of attention. As Daniel Brook points out in The Trap, 28% of Americans have considered starting their own business. And yet only 7% actually do.
What holds them back? The lack of a social safety net. A friend of mine, a brilliant young technologist who's been featured everywhere from PBS to Salon, stayed in academia and the corporate world while all of her friends were starting companies and getting rich. Why? Because she couldn't afford to lose her health insurance. Between skyrocketing prices and preexisting condition exclusions, it's almost impossible for anyone who isn't in perfect health to quit their job. (I only managed because I was on a government plan.)
Anyone with children is also straight out. Startup founders tend to be quite young, in no small part because no one can afford to support a family on a startup founder's salary. But if we had universal child care, that would be much less of an issue. Parents would be free to pursue their dreams, knowing that their children were taken care of. And universal higher education could let parents spend their savings on getting a business started, instead of their children's tuition. Plus, it'd give many more kids the training and confidence they needed to start a company.
And those large private wealth funds that result from growing inequality? A real problem for startup founders is that they're too large. It used to be that you could borrow a couple thousand dollars from friends and neighbors to get your business off the ground. Nowadays, they're too busy trying to make ends meet to be able to afford anything like that. Meanwhile, those large wealth funds I mentioned are now so big they can only afford to invest in multi-million dollar chunks -- much more than the average founder needs, or can even justify. And the large investments come with large amounts of scrutiny, further narrowing the recipient pool.
But imagine if the government provided a basic minimum income, like Richard Nixon once proposed. Instead of having to save up (increasingly difficult in a world in which the only way to survive is on credit card debt) or borrow money to stay afloat, you could live off the government-provided income as you got things started. Suddenly having to quit your job would no longer be such a huge leap -- there'd be a real social safety net to catch you. (Not to mention if those labor laws some people want to loosen required your old job to take you back if things didn't work out.)
Of course, there is some truth to the standard proposals. Some startup founders are encouraged by dreams of financial security, and high taxes can make that dream more elusive. And complex labor regulations can make it difficult to get new companies off the ground. But it's not an issue of whether we should have taxes or labor laws -- it's an issue of how they're targeted.
Estate taxes on inherited fortunes would have basically no impact on startup founders, but could go a long way to funding a social safety net. And since most startups are acquired as stock, income taxes are basically irrelevant -- it's really capital gains tax that gets applied. There's no reason the government couldn't apply a lower capital gains tax to startups that get acquired than they do to the shares of publicly-traded companies that large investors trade.
The same is true for labor laws: preventing large companies from firing people at random can provide some much-needed stability to their lives, especially if they're saving up money in the hopes of going into business themselves. But there's no reason such laws also have to be applied to small startups, where the company is more likely to go out of business than to fire you.
Look at social democratic Europe, where these policy prescriptions have been tried. While there's much less of a culture of entrepreneurship and only 15% of Europeans think about starting their own company, nearly all (14.7%) of them actually go ahead and do it.
The fact is, if governments really want to promote startups and the economic innovation they bring, they shouldn't listen to the standard refrain of cut taxes and deregulate. They need to start rebuilding the social safety net, so that their citizens know that if they go out on a limb and try something risky, someone will be there to catch them if things don't work out.
Thanks to Daniel Brook's book The Trap: Selling Out to Stay Afloat in Winner-Take-All America for suggesting this line of argument and providing the statistics.
June 05, 2008
Redmonk
Nano-Power
Ok, these may be really really stupid or really really practical ideas — so no comments starting with “Steve you’re an idiot!” — but I was thinking today about ways to generate power around the house.
What if electric power could be practically generated and distributed by micro or nano devices embedded in motion-generating fixtures around a house? Opening doors, drawers, running water from faucets - there’s a lot of energy that perhaps could be captured. I doubt it would be a lot of power, but enough, perhaps to charge small devices?
Crazy? Interesting? What do you think?
June 04, 2008
Redmonk
DiSo: XRDS on the move
Over on the DiSo Project blog, I’ve posted about some movement in the XRDS-Simple space over the last week. Next steps: We need to identify generators, parsers, and service definitions, and start working on interop.
May 30, 2008
Kevin Reid
Re: Zippers
For those wondering about the problem I was trying to solve with a zipper: After all the advice and some consideration, I decided to go with “option #2”: a whole-tree processing step which adds equal labels to corresponding variable uses and bindings. I’ve implemented all of the analysis I described in the original post using this system.
(However, the code isn't quite finished (as I discovered some problems with a different aspect of it) so it's not in any public repository yet.)
May 29, 2008
The Mocking Eye
Political Spectrum Through Fiscal Preference
I was thinking today that most political affiliation to parties and left/right wing, can be concisely encapsulated by looking at the top Fiscal Priority of an individual. This seems to be able to identify parties and lables that otherwise elude simple description, since they contain a variety of various other views.
For example, the US Democratic Party favors government spending maximized towards social programs. The US Republican Party favors–at least since Reagan–government spending maximized towards defense. The US Libertarian Party favors decreased fiscal spending in all categories, and the US Green Party of course favors maximal environmental spending. I’m not quite up to date on the Israeli party zeitgeist, but this same methodology can be applied to them as they were when I left in 2000.
People often complain of the blurring of distinction between parties and labels, as do I. Usually such an affiliation tells you little about the person identifying as something or other except who they will likely vote for. Now, however, you will be able to understand their fiscal priorities. Anecdotally (that is, as far as I can stretch this thought exercise without actual data and polls), this is the only policy preference that is tightly-coupled to labels.
Bypass the screen of parties and cumbursome ideology: just ask people for what the top spending priority should be for the gov’t, realistically*.
* This means that although a Republican may want to spend the most on, say, Education, their prefered foreign policy would require heavy military spending, thus in reality, they would maximize defense spending. People are not always honest with themselves about these things, so you have to present this caveat explicitely to attempt to ellicit accurate replies.
Share ThisMay 28, 2008
Christopher Schmidt
Geographic Queries on Google App Engine
I made a post over on the MetaCarta Labs blog about doing geographic queries on App Engine:
???????? ????? ????????In this way, I was able to put together a geographic bounding box query, on top of Google App Engine, using a Geohash-like algorithm as a storage format, and use that query to power a FeatureServer Demo App Engine application, doing geographic queries of non-point features on top of App Engine/BigTable. Simply create a Geoindex object of the bounding box of your feature, and then use lower-left/upper-right points as bounds for your Geohash when querying.
– Geographic Queries on Google App Engine
Probably of interest to some of the people who read this.
May 27, 2008
Redmonk
XRDS Simple for Movable Type
DiSo is built on the idea of implementing social networking concepts on existing standards where possible. One of those standards is XRDS Simple, a service description and discovery format that is a part of several recent and upcoming standards including OpenID and OAuth. This weekend I decided to build a generalized service for Movable Type that implements XRDS.
XRDS Simple for Movable Type (version 0.1)
XRDS Simple for Movable Type implements XRDS Simple service discovery. New web services like OpenID and OAuth are encouraging the use of XRDS (eXtensible Resource Descriptor Sequence) (and specifically the new, simplified version) for service discovery.
XRDS Simple for Movable Type is a faceless application that gives plugins a way to register services to be advertised in the discovery document, similar to how plugins can register new streams with Mark’s Action Streams plugin.
Registering A Plugin’s Services
Sample configuration YAML:
xrds_services:
openid2:
type: http://specs.openid.net/auth/2.0/server
# media_type:
uri: http://endpoint.example.net
namespace_id: openid
namespace_uri: http://openid.net/xmlns/1.0
priority: 10
# local_id: http://redmonk.net
local_id_handler: sub { "http://example.com" }
A plugin registers its services with XRDS Simple by including an xrds_services section in config.yaml. In that section are any number of service definitions, each listed under a unique identifier.
Within the service definition are a number of items which generally follow the XRDS Simple spec[1]. Of note are the namespace_id, namespace_uri, and local_id_handler.
Each service can register a namespace to be included in the xml declaration of the discovery document. The namespace_id and namespace_uri parameters are used to construct an xsmlns: attribute, like:
xmlns:openid="http://openid.net/xmlns/1.0"
XRDS also supports the concept of a local id - an identifier for this site or user on the service the discovery document points to. The local_id parameter will be added to the generated document in the service definition. Sometimes the local_id will need to be set programatically. In that case, pass in a coderef as local_id_handler.
local_id_handler: sub { "http://example.com" }
What’s it good for?
XRDS is developing as the discovery mechanism for a new generation of social applications. Eventually, Movable Type’s OpenID server and delegation could be changed to use this simpler yaml-based configuration, and let the plugin generate the appropriate XRDS document, instead of generating the document itself. As Movable Type begins to offer OAuth-based services, those too can be listed - per the OAuth Discovery spec.
Check it out
As this is a faceless application, and there are currently no services in MT using this code, this is really developer-preview code. Download and peruse the plugin - it’s very simple. I’d be interested in feedback on the code and the YAML format as well. Upcoming features include support for multiple Type and MediaType elements (per the spec), as well as support for passing a coderef for parameters instead of static values.
Download
Download XRDS Simple for Movable Type or check it out from the DiSo repository.
Fascinating Feynman
I have no idea how to categorize this post from longnow.org about Richard Feynman, a fascinting and personal look at one of the giants of modern physics. Find yourself 20 minutes of pressure-free time and read it, then read it again.
May 21, 2008
Christopher Schmidt
Wherecamp2008: Ideal Conference-Going Experience
Wherecamp is the closest thing to a perfect conference going experience I have had in my history of going to conferences. The organizers — Ryan and Anselm — did a great job of pulling everything together, but I think that the single entity most responsible for making this happen was Google, and their representatives on the ground, Nate and Dusty.
Key things to good unconferences:
- Constant stream of food. Geeks like to eat, and eating keeps us happy, healthy, and wise. Snacks are great, both healthy and unhealthy, and drinks and so on are also good. I think this, more than anything else, is the kind of thing that a well-funded social event should try to have. The meals at Google were also great, and having the space we did for serving was super useful for avoiding frustration at lines. In general, for the number of people who were present, Google’s facilties were significantly better prepared for serving people than my experience at Where 2.0 and FOSS4G events.
- Big open space for organization, small cozy meeting spaces for sessions. One thing that repeatedly came up was that the conference spaces we were in for sessions were full — and, believe it or not, I find sitting on a makeshift chair in the back of the room *great*. The participation in the sessions was better than at any other unconference I’ve been too, because of the close-knit situation. Occasionally things were a bit too small — the Processing talk was one example — but in general, it was a good thing to be in close quarters.
- Great wireless. Coverage of the entire space was perfect, including outside, and given that there were 300 people sitting around banging on it, that was far more impressive. I’ve seen networks fall apart under significantly less load, even at Barcamp Boston 2, at MIT.
- Open Space: The central auditorium and outdoor areas were great places to chill out and enjoy your time. Not having to be stuck in a hallway, or running back and forth between sessions, was a wonderful end result, and the tables/chairs/etc. setup was just great.
The last thing that can make or break any conference is the people, and I couldn’t have asked for any better. I met and hung out with a bunch of Python Geo hackers — more so than I ever have in the past. We stayed up late, hacked all night (in my case, literally) and resolved a ton of issues. I got to hang out with some of the people that I really enjoy spending time around, and meet some new ones.
Thanks to all of you who made this weekend totally worth the trip. Special thanks to the organizers, and Google for being such a great host.
Next up is FOSS4G, I hope; I hope that I can take what I learned here and make an effort to turn FOSS4G into (at least partially) the event that Wherecamp was so successfully.
May 20, 2008
Redmonk
Phusion on mod_rails
Well, as you’ve stated, we believe that experienced Ruby on Rails developers don’t consider deployment to be a problem; probably annoying at most. They’ve automated/standardized the deployment process for a great deal with Capistrano scripts. Newcomers however see deployment as an insurmountable problem. Consequently, we end up having a situation in which the people who can solve the problem either don’t see a problem or aren’t motivated enough to fix the problem, while the people who do see the problem cannot solve it.
Furthermore, from our personal experiences, newcomers usually want to see immediate results. It’s very demotivating for those people to continue using Ruby on Rails if they have to manage tons of obscure configurations and clusters to ‘just successfully deploy an application’. This leads to very intelligent conclusions, such as that “Ruby on Rails sucks”.
This is exactly what killed my nascent interest in ruby-on-the-web. I continue to love ruby-as-intelligent-shell-script. I may have to give ruby-on-the-web another go.
Morbus Iff
Include, and style, a Drupal 6 teaser on a node's full view
In the Drupal content management system, a "node teaser" is small bit of content used to encourage you to "read more" of the post. Drupal can set the teaser to the entire length of the post (typically used for blogs where you don't need extra click-through), or can automatically generate the teaser to a specific character length. In the past, you could also manually generate teasers by including <--break--> in the node's body. In Drupal 6, manual teaser definition has been improved with JavaScript wizardry, along with a new checkbox: "Show summary in full view".
But there's a small problem with the use of the word "summary". Generally, when a Drupal teaser is included in the node's full view, it's because it's the introduction of the node itself, not necessarily a teaser or summary of the entire body. Over at gamegrene.com, a node's teaser is, in fact, a summary of the node, and is also displayed on the full view itself. It's not the first paragraph of the article but, rather, is styled differently to provide an overview of what you'll be reading. IBM uses the same model at their developerWorks.
If you placed a "summary" at the beginning of the node's body, unstyled, readability would tend to suffer - you'd have the summary (node teaser), and then, theoretically, the introduction (node body), with no clear indication that two different types of content, with two different purposes, are being served.
As I've been working on moving Gamegrene to Drupal 6 (in time for Dungeon and Dragons 4th Edition, coming June 7th), I had to solve the problem of: how do I theme the teaser differently than the body inside node.tpl.php? By the time the template gets the node data, only $body and $content exist; $content only contains the teaser (for list views) or body (for full views). The teaser never exists in a node's full view as its own variable.
To solve this and get the same view as seen on IBM's developerWorks, I used themename_preprocess_node() to detect if a teaser has been manually set and that the "Show summary in full view" checkbox has NOT been enabled. When that checkbox is checked, Drupal automatically adds the teaser to the node's $body (or $content) - it treats the teaser as the introduction to the post, not an actual summary of what you're reading:
function phptemplate_preprocess_node(&$variables) {
// we like to display teasers on the node view pages in a different style,
// but only if they were NOT set to "show summary on full view" (which seems
// backward, but the implication with that checkbox is that the teaser is
// PART of the node's body, instead of an actual summary of the entire
// node's body). if a node's unbuilt body starts with <!--break-->, then
// a teaser has been manually set, and "show summary" is not checked.
if ($variables['page'] == TRUE) { // only do this on full page views.
$node = node_load($variables['nid']); // we reload the node because
// by the time it gets here <!--break--> has already been filtered out.
// this if logic stolen from node.module's node_teaser_include_verify().
if (strpos($node->body, '<!--break-->') === 0) {
$variables['style_teaser_differently'] = TRUE;
$variables['teaser'] = check_markup($node->teaser, $node->format, FALSE);
}
}
}
Note that the extra node_load() is nothing to worry about - since the node has already been loaded earlier in this execution, node_load() will happily return a cached version, saving us any performance concerns.
Now, it's just a matter of displaying it in node.tpl.php:
<?php if ($style_teaser_differently) { ?>
<div class="node-summary"><?php print $teaser; ?></div>
<?php } ?>
Comments and concerns? Note that, for my particular needs, I wanted this entirely in a theme - I'm not changing data or its structure, merely its display, so doing this sort of stuff in hook_nodeapi() with a module's overhead would be a little much.
Redmonk
OpenId and Oauth, An Introduction
These are the slides from a presentation I gave to about 80 developers at work at a recent day-long training event.
May 19, 2008
Aaron Swartz
The False Consciousness Falsehood
American intellectual life has a large number of ways of responding to an argument without actually addressing its substance -- namecalling in other words. You can say that someone is "blaming the victim" or spinning a "conspiracy theory" or "assuming people are stupid" or that they're subject to "false consciousness".
Most of these are kind of transparently silly, but even otherwise smart people seem to think the false consciousness charge has some heft to it. The argument is never fully spelled-out, but the argument seems to be that to think that people are systematically mistaken about their own interests is the kind of crazy idea that only vulgar Marxists would believe and, furthermore, it requires assuming that people are stupid and explaining how you've been able to see past the illusion.
Well, I'm personally not under any illusion that providing a rational explanation is going to stop people from leveling this charge, but I figure one ought to, if only to set the record straight.
Let's begin with a parable -- a simplified case that will at least establish whether some of these arguments are logically true. Imagine a new regime comes to power that decides to imprison everyone with red hair. They insist that there is nothing amiss about this -- they were elected democratically, and furthermore, everyone imprisoned is still allowed to vote. But inside the prisons, they only permit limited contact with the outside world. Most prisoners only watch the one prison-provided news station which is systematically biased, constantly suggesting that the Purple Party is in favor of additional rights for red-haired people while their opponents, the Yellow Party, just used the red-haired issue for pandering. (Anyone who's watched, say, Fox News discuss black issues will know how this is possible.) The result is that when election time rolls around, the majority of red-haired prisoners vote for the Purple Party candidate who gets into power and provides no new rights for them.
Call it false consciousness or not, I think it's perfectly reasonable to look at this situation and say while the red-haired prisoners are not stupid, they are systematically mistaken, which is leading them to act against their own interests. If they knew the truth they would vote for the Yellow Party, the party which wants to take steps to get them out of prison, instead. Furthermore, it's possible to imagine that there are some prisoners who, through one means or another, have learned this and thus are able to see this situation while the other prisoners do not. (They try to tell the other prisoners what's going on, but they keep getting labeled conspiracy theorists.)
Now obviously vast portions of America are not imprisoned. But most people do get their news from a small number of sources and I think everyone would agree that, in one way or another, these sources are systematically biased. (You can argue about which way they're biased or whether it makes a difference, but I think it's pretty clear that all the major news sources share a general conception of what is "news" and what isn't.) So why is it so implausible that something similar is going on?
The major difference between the two scenarios is that in the first, people were basically forced to watch the biased news, while in the real world they have lots of other alternatives. But I'm not sure this matters as much as it might seem at first.
First, most people have busy lives that don't revolve around the news or politics and thus are going to get the news in the most convenient form they can. For most people, this is typically television or the newspaper. But starting a new television station or newspaper is very expensive, especially if you want it to have wide reach, and the only projects that can get funding and advertising are those that buy into at least some of the systematic biases. So for most people, there simply isn't a better alternative when it comes to the formats they want.
Second, even if someone gets their news from the Internet or another source where getting started is less expensive, they may not know about the alternatives. If you grew up with your parents reading the New York Times you may simply live your life checking in on nytimes.com, without ever stopping to wonder whether the news you were getting was systematically biased and whether there was some more preferable alternative.
Again, just as there was no way for the prisoners to know they were being lied to, it's not really reasonable for the average person to figure out that they're getting biased news if the only news they read comes from biased sources.
Now I'm not arguing here that this idea is true (that would require more real-world evidence), merely that it's possible. The fact is that we live in a world where most people get their information about what's going on from a very small number of sources which tend to report largely the same things in the same way. This seems like a rather important fact of life and I think we ought to stop dismissing suggestions that it might have some negative effects on people out of hand.
May 17, 2008
Christopher Schmidt
Cool Processing Stuff
Notes about Processing, from processing.org, and other related things.
“Drawing images is not about telling which pixels to turn on or off, it’s about telling to draw lines.”
In addition to processing being pretty, you can draw the entire map into any unit you want. Drew a bunch of lines out in state plane projected unit system, and then cropped image.
Drawing all of the streets in King County plus all of transit lines
4d expansion of transit data — time out of the map on photo
Processing is great at generating static images; looping over a dataset and drawing lines is what processing is for.
Demoing a bunch of demos of processing from Tom Carden; cabspotting, travel time tube map, etc. etc.
Processing.js: Processing in JS, by John Resig. Uses “Processing.js is like developing Processing on a 400mhz celeron”
Source code to London Tube Map is available; Processing makes source code available by default, Google for “powered by processing” for examples.
Stamen typically does Prototype in Processing, final client in Flash.
Processing is Java; it should be possible to pull in Java libraries; can import .jar files, can drag in data (images, etc.); loading data is a one liner.
NodeBox: “Processing in Python”
Obsessing.org: Processing.js GUI.
Same processing source code in browser and applet using Processing.js
Context Free Art: here
The default mode of processing is:
* Setup
* Draw function that’s called as much as possible
Going to publish cabspotting example to wherecamp wiki
Brandon demos OpenStreetMap data rendered on the fly with processing data.
NodeBox demo using a springgraph. Proxmiity fixes of local bluetooth data.
Processing is 4-5 years old now
Collaboration is typically “Copy an example, create your own, keep going”
May 16, 2008
Christopher Schmidt
Geohacking This Evening
Tonight, starting around 7:30, there is a plan to descend on The Dubliner, in Noe Valley, San Francisco, for beer and hackery. If the plan changes, I’ll update twitter. If you want to join us, please:
- Comment here
- Email me (crschmidt@crschmidt.net)
- or Text or call me (603.264.2294)
The Mocking Eye
Grahpically Representing a Programming Language
I have been thinking about taking some programming language and making some sort graphical interface akin to StarLogo TNG but I may be approaching it too a posteriori. That is, I’m assuming a graphigal, i.e. spatial, representation of it would be necessarily good. But I think that follows from the principles of UI design: user interfaces are all about metaphors and similes, and the closer they are to experiences humans have evolved to handle well (like manipulating physical objects) the easier it is for humans to grok.
I thought long and hard about using Python but its syntax and semantics just don’t respond to spatialization very well. That’s when I thought of using Haskell and it might be just perfect for it.
When I was trying to model Python spatially I ended up with perverting it into some sort of data structure oriented language. That is, the graphical objects combine into bigger objects which represented data structures.
Haskell on the other hand would let me turn just about every user defined function into a physical object, and most built-in functions into a physical idiom. Rather than mere construction of playing blocks, I will translate operators (such as cons) into actions the user performs in the design environment.
May 14, 2008
Redmonk
Distributed Social Networkers
Yesterday the news came out that Chris Messina, my DiSo co-conspirator, and Will Norris, author of the WP-OpenID plugin, are joining Vidoop to work on DiSo full time. This is excellent news, for Chris and Will obviously, but also for the ideas and ideals that DiSo represents.
DiSo’s stated goal is to be: “an umbrella project for a group of open source implementations of these distributed social networking concepts.” At the core, DiSo represents an ideal where online “social networking” is not limited to silo sites, where you can enjoy rich interactions with others in that silo at the expense of interacting with those outside. It also means that your online social identity is not a game of tetherball - where your idenity can follow you around some, but it’s still tied to that one provider (I’m looking at you, Google).
No, in the ideal that Chris, Will, Stephen Weber, myself, and many others envision, your identity, your friends/contacts, your interactions can be managed from a space that you control - your own site. This may be a weblog - WordPress is the DiSo Project’s first platform - or some other type of site, but it’s your site. And by building components on common internet standards (Microformats, OpenID, Oauth, XMPP, etc) your home on the web can join a larger network of people, regardless of platform, including but not limited to those users in the silos.
This is a large goal - a grand scheme if you will - and it’s being built in the open, piece by piece, by a group of developers who see the larger picture and are contributing their time, knowledge, and skills. Scott Kveton is someone who has been around the Open Web block and is excited about DiSo; by bringing Chris and Scott on board, he and Vidoop are investing in an open future that is looking bright for all of us.
Christopher Schmidt
Wherecamp, here I come!
I’m off to Wherecamp tomorrow. I’ll be in town and have no plans yet as of Friday, and I’ll be at wherecamp all weekend, planning to spend the night hacking at the Googleplex.
Who’s going to be there? Anyone interested in meeting up on Friday, or doing any hacking ahead of time or during?
Aaron Swartz
Tectonic Plates and Microfoundations
In 1915, Alfred Wegener argued that all the continents of Earth once used to fit together as one giant supercontinent, which he later named Pangea. As Wikipedia summarizes:
In his work, Wegener presented a large amount of circumstantial evidence in support of continental drift, but he was unable to come up with a convincing mechanism. Thus, while his ideas attracted a few early supporters ... the hypothesis was generally met with skepticism. The one American edition of Wegener's work ... was received so poorly that the American Association of Petroleum Geologists organized a symposium specifically in opposition.... ... By the 1930s, Wegener's geological work was almost universally dismissed by the scientific community and remained obscure for some thirty years.
Today, of course, every schoolchild knows about Pangea. But for a long time the theory was dismissed, not because it lacked evidence or predictive power -- it explained why the shapes of the continents fit together, why mountain ranges and coal fields lined up, why similar fossil were found in places separated by oceans, and so on -- but because Wegener had no plausible mechanism.
A similar problem happens in the social sciences. Paul Krugman recently noted that while Larry Bartels (in his new book Unequal Democracy) provides solid, convincing evidence that Republican presidents systematically preside over slower growth and increasing inequality, most social scientists don't believe him because we haven't yet identified the mechanisms. Krugman:
Now, I'm a big Bartels fan; I've known about this result for quite a while. But I've never written it up. Why? Because I can't figure out a plausible mechanism. Even though I believe that politics has a big effect on income distribution, this is just too strong -- and too immediate -- for me to see how it can be done. Sure, Republicans want an oligarchic society -- but how can they do that?
Bartels, for his part, argues that providing the mechanisms isn't his job -- his goal is to highlight the phenomena and encourage many others to research the mechanisms:
How do presidents produce these substantial effects?
One of my aims in writing Unequal Democracy was to prod economists and policy analysts to devote more attention to precisely that question. Douglas Hibbs did important work along these lines ... He found that Democrats favored expansionary policies ... while Republicans endured and sometimes prolonged recessions in order to keep inflation in check. (Not coincidentally, unemployment mostly affects income growth among relatively poor people, while inflation mostly affects income growth among relatively affluent people.) In recent decades taxes and transfers have probably been more important. Social spending. Business regulation or lack thereof. And don't forget the minimum wage. Over the past 60 years, the real value of the minimum wage has increased by 16 cents per year under Democratic presidents and declined by 6 cents per year under Republican presidents; that's a 3% difference in average income growth for minimum wage workers, with ramifications for many more workers higher up the wage scale. So, while I don't pretend to understand all the ways in which presidents' policy choices shape the income distribution, I see little reason to doubt that the effects are real and substantial.
When it comes to addressing such arguments more generally, the most famous commentator is Jon Elster. In his classic article "Marxism, Functionalism, and Game Theory", he insists:
Without a firm knowledge about the mechanisms that operate at the individual level, the grand Marxist claims about macrostructures and long-term change are condemned to remain at the level of speculation.
(To be fair, Elster doesn't make this as a general argument, but his vehemence has led some of his followers to suggest that it is.)
To be clear, I think discovering mechanisms is important work. All I'm arguing is that it shouldn't be a necessity for believing in a theory. Instead, I believe it's an irrational side-effect of an emotional distaste for gaps in knowledge.
As evidence, let me note that such demands for mechanisms never go more than one level deep. Nobody has ever said, "Well, your theory that people are motivated by greed is all very nice, but I just can't believe it until you can explain how greed is manifested in the brain." Neuroscience is obviously the microfoundation of psychology, but psychological theories are regularly accepted without neuroscientific microfoundations.
In general, it seems like such commentators support a double-standard. Theories with mechanisms should be judged by their fit with the evidence and predictive power. Theories without mechanisms should be judged by the evidence and predictive power and whether you can think of any plausible mechanisms. I don't see how this can be justified. There's no reason mechanism should be privileged in the assessment of knowledge; things are true or false, even if we don't know why they are true or false.
Indeed, it we typically only investigate the causes of phenomena once we're convinced that they exist. (Elster admits as much in Explaining Social Behavior, noting that establishing a phenomena's existence is the first step towards explaining it.) So let's stop making the mistake of not believing things are true because we don't know how they happen.
The Mocking Eye
The Shahadah (Islamic Declaration of Faith) in Python
A moment of fancy on IRC led me to concoct the following:
((len([i for i in __globals__ if i is not god and type(i) is type(god)]) == 0) and (god.prophet == muhammad))) == True
This is the Shahadah represented in Python.
Programming Language Wars are frequently religious, I thought why not take it a step further.
Share ThisMay 13, 2008
The Mocking Eye
Checking in to Brightkite
I’ve been using Brightkite for several weeks now, and I rather enjoy it. What is it? Well, it’s a GPS-less location tracking service, of sorts.
In its actual usage it is much like Twitter. You register your cellphone , and whenever you arrive at a new location you can ”check in”. You just SMS your current address to the service, and it registers that. You can also search for businesses and pick the correct one. It levereges the Google Maps API for location degection based on the address you provide.
Once you’re checked in to a location, you can post notes, little messages in the same vein as twitter, which will be listed both globally, and within the location’s stream. You can also send in pictures to a location.
Of course, you can keep track of where your friends are and their various posts. There is some rudimentary permissions control. There are three groups: Public, Friends, Trusted Friends; and you can decide which can see your exact location, or only the city you checked in, and which can see the notes you post.
The system notifies you of people checked in nearby, if you so wish, within a radius you specify.
My favourite feature of the whole thing is its replication to Twitter. That is, any post you make to Brightkite also appear in twitter, with a little http://bkite.com/xxxx URL appended, which when clicked shows your location on a map.
As I was checking in at my work place I had an idea. So I took a screenshot of their check-in button (I hope they will forgive me that transgression). Now I can display it on some site, and garnish it with a check-in link to whatever location is related to that site. For example, on a conference page, you can just click that button to indicate you have arrived, and everyone at the conference on Brightkite will be notified. Here’s an example of it checking you in at MIT (requires Brightkite account):
Cool, eh?
Brightkite brings location-aware augmented reality one step closer for those without convenient GPS.
Want an invite? I’ve got some left. Leave a comment somehow containing your email (if I don’t know it), and if I like you enough, I’ll send you one!
Share ThisAaron Swartz
Simplistic Sociological Functionalism
(I thought I should talk about the other form of functionalism for a change.)
Often sociologists notice a pattern in which certain attributes of a social system fits well with a particular social structure. To take an example I have at hand, Rosabeth Moss Kanter notes that because a secretary has access to facts that could embarrass her boss, it's convenient for the boss that the secretary is entirely dependent upon him for wages and status.
Unfortunately, these claims are often phrased as saying X causes Y. Here's how Kanter does it:
The possibilities for blackmail inherent in [a secretary's] access ... to the real story behind the boss's secrets ... made it important that she identify her interest as running with, rather than against, his. Thus, forces were generated for the maintenance of a system in which the secretary ... was to find her status and reward level dependent on the status and, hence, success of her boss. (Men and Women of the Corporation, 82)
Note that, although she is unusually careful to hedge her comments ("made it important", "forces were generated", "maintenance of a system") Kanter is making a particular historical claim here: the secretary could blackmail, which pushed the boss to tighten control. But this is not the type of claim that Kanter, who's research consisted mostly of direct observation of present-day offices, is likely to have any real evidence for.
Making such claims is problematic, both because most sociologists don't really know whether they are strictly true, and because they lead Jon Elster to show up at your house and yell at you for hours. But both problems can be easily avoided: simply rephrase such comments to describe the phenomena as effects rather than causes.
Instead of saying a secretary's ability to blackmail leads bosses to tighten their grip, simply note that the boss's tight grip has the effect of weakening the secretary's ability to blackmail. You get all the same points across and nobody gets hurt. See? Easy.
May 12, 2008
Aaron Swartz
How to Fix the News
Newspaper circulation continues to decline. The top-selling paper in the country, USA Today, distributes only 2 million copies a day (half, no doubt, placed outside hotel room doors). Around the same number, with an average age of 71, watch The O'Reilly Factor nightly, with the number decreasing as the audience dies off. Everyone quietly concedes the news industry is dying. It's the Internet's fault, they all assure us.
But what if it wasn't? The other day I heard a news program that was so good that I wanted to listen to it again. And I'm not alone -- all my friends have been talking about it as well. And while I don't have exact numbers, it seems as popular as any one of those other news outlets. That show? The This American Life episode on The Global Pool of Money -- a comprehensive explanation of the housing mess.
There were three things about the show that made it stand out from the rest of the news pack:
It believed in the intelligence of its audience. It didn't try to pander with sex or disasters or quick cuts. It took a serious news story and investigated it thoroughly for a full hour, with only one break. And it didn't try and dumb any of it down -- it explained the whole thing, from top to bottom.
It didn't assume you already knew the subject. Most news stories on important topics are incomprehensible to the average person who doesn't know much about their topic. Here's a quote from a random news story about the housing crisis: "They said financial institutions have been unwilling to expose themselves to the mortgage market, and lenders are hesitant to lend to risky borrowers in a declining house price market after the subprime meltdown." Unless you've been following the story (like the reporter, presumably) do you really know what that means? TAL instead assumed you knew nothing and explained every component and term so that you actually had a picture of what was going on.
It was done in an entertaining and conversational tone. It didn't treat the news as some important series of facts that had to be seriously conveyed to you. It treated it as something interesting they wanted to tell you about, a story that involved real people's lives (who you got to hear from at length) and was full of genuinely interesting pieces. Look at that news quote above one more time. Can you really imagine someone sitting down and saying that with a straight face?
At first these things may seem contradictory -- how can you believe in the intelligence of your audience while assuming they don't know anything? how can you be entertaining and yet still explain a subject? -- but the more you think about them you see how well they fit together. Being intelligent doesn't mean you're