Computers are absolutely incredible achievements of human ingenuity. Computers are regions of the universe that are under a much tighter level of human control than any other physical arrangement of matter into a technology yet conceived. Inside of microchips, electricity is serving a purpose directly prescribed by a human at a keyboard.
Each computer is a special place in the universe. Computer chips are arrangements of matter that behave so predictably that numbers and mathematical operations can be encoded in patterns of their state. Every active process has a corresponding physical mapping on the microchip expressed as a pattern of semiconductors that are on or off. We see graphics on arrays of pixels, what we do not see are the datastreams coming from immense grids of switches fluttering on and off in programmed informatics. The ability to hold onto huge strings of instructions and do varying operations on those instructions has transformed mathematics into a firehose of computation, so many operations that you can paint entire 3D worlds.
External computation combined with digital networking is the most powerful technology suite humanity has ever created, and already realizing that we are enveloping the world in, to use Luciano Floridi’s term, the infosphere. None of this though is likely news to you. It seems to be common understanding that computers and computer systems are immensely powerful. Much ink and many pixels have been spent discussing the opportunities and pitfalls of an increasingly cybernetic world. Math and programming are understood as extremely influential forces today, with Lawrence Lessig declaring that code is law. With people spending increasingly large amounts of their time in digital platforms code has a more immediate impact on many lives than law now.
If I know something of this, then why is it that I do not really know how to program?
I have been trying. And I’ve been finding joy and frustration. Coming from a graphic design background I am in a foreign place when trying to understand and manipulate programs. I was always fascinated by computers and assembled PCs and hacked around, but I never learned programming. I was math phobic and thus I was programming phobic. That was the narrative I internalized. Only now am I realizing that I can learn math and programming. I can cobble things together, but I do not understand the concepts, I do not feel comfortable, I am always using duct tape, never a nice hand tool. I am on a quest to learn myself and make it easier for everyone else to learn.
I believe that everyone deserves to know how to think systematically and execute their intention in the form of computer programs. The ability to implement formal logic is part of the new literacy.
“Imagine that children were forced to spend an hour a day drawing dance steps on squared paper and had to pass tests in these ‘dance facts’ before they were allowed to dance physically. Would we not expect the world to be full of ‘dancophobes’? Would we say that those who made it to the dance floor and music had the greatest ‘aptitude for dance’?” Mindstorms (Papert, page 43)
My childhood exposure to math was a series of facts and modules that I needed to repeat exactly for...some reason.
I was raised in a DIY family, with my dad being a competent carpenter and furniture maker. Assembling materials in 3D space is something I have always had a sense for, yet I did not manage to connect the theory of school geometry with practice. I was unable to make the analogy leap from scribbling symbols on paper to genuine interalization of the concept.
So I carried through my life that I was "bad at math" even though I would routinely use logical reasoning to fix broken computers and help my dad with construction and business. I could not see that programming was a different sort of math, I thought I just couldn't do any of it. I became a proficient user of the Adobe suite and settled into an ability to make static 2D designs. Recently though I managed to branch out and become a UX consultant for a startup Collaborate.org Now I find myself mocking up user interfaces and the corresponding logic that goes with them, and I'm finding that my tools were not designed to bridge the gap between graphic design and code. There are systems that allow the creation of interactive mockups, but I have become obsessed with the visceral input that using a digitizer pen to sketech ideas and comment on others' work allows.
In my third semester of CCT I learned how to make images in Processing and found that was the music for me to understand why I would do math, and what math can let me do. I was not unable to do math, I was simply never shown the right connection to spark my understanding.
Now I find myself as someone who is grasping the concepts of programming, but has a hug eamount of experience, and an unashamed bias toward, using systems that provide the ability to manipulate on screen elements directly and offer instant, constant feedback. I want to push code around literally.
But why? Why can't I suck it up and learn? I can, and I am, but first we need to understand why I never learned to program in the first place.
In Don Norman’s “The Design of Everyday Things” he provides ‘principles of good design,’ which are:
Traditional “typed statement” programming environments live up to none of these criteria. In traditional programming it is often very difficult to determine relationships between operations and results. The feedback is not continuous and because of syntax constraints often a program being modified is not in a state where you can see the results. Systems like spreadsheets are better, since they offer easier to choose alternatives and usually allow you to see the results with just a click, providing a nice feedback loop. Norman concludes that with spreadsheets “it felt as if you were working directly on the problem, not on a computer.”
This is not just for interactive systems. I have embedded Bret Victor's example of a re-designed Social Network Analysis paper, and I urge you to play with it. You will see that the outputs of the desribed algorithms and the picture of the concept is much more apparent when you can just interact with it. We are in the web era now, there is no real reason to subject people to equations alone when we can provide support structures that can help in higher-order comprehension.
I think that it should be much simpler to create the kind of rich interactive media that Victor is demonstrating. We need a graphic design tool for the modern computer era.
I would consider myself a graphic designer so I am well versed in pushing shapes around the screen and now with a tablet computer, drawing curves. In learning about programming and attempting to make my own things I find myself yearning for direct visual manipulation mechanisms. In my primitive state of thinking, I imagine something like Adobe Illustrator if the whole script of the canvas could be accessed. I want to make an object and copy it with a right click. I want to link functions together spatially on a page as I would elements of a graphic design. I want to make mathland a very real idea, instantiate it in computer graphics and give people the ability to get literally hands on with concepts.
I have a few working research questions that are guiding what I am trying to learn in order to actually understand the problem space:
I have used MaxMSP and looked at VVVV which are diagrammatic programming environments. It is really fascinating to manage control flow by connecting “wires” between functions, gates, variables, databases and trigger events. I find them compelling since it presents a level of interactivity that appeals to my mind, and I have even found that these styles of programming work pretty well collaboratively. Two people can look at the diagram at once and both be looking for different bugs or next steps. However, neither of these systems use different icons for different kinds of concepts, so it results in a giant tangled mess that you still have to zoom into in order to actually make sense of.
There are modular systems like Scratch and Hopscotch but these are geared primarily at children and so come with unnecessary cuteness quotients and limited feature sets. I am not only interested in making games, but want to sketch entire interfaces and databases, and I think kids should be able to head in that direction too if they find themselves moving that way while working.
There is a modern version of Alan Kay's Smalltalk called Squeak, which functions as an Integrated Development Environment (IDE). However the interface is based primarily on the WIMP paradigm and has no pen integration. It seems to have a plethora of icons and controls.
Digitizer pens are critically under-utilized input devices. The systems I have found, along with my personal experience using my Microsoft Surface Pro 4, show a world where using a computer is much more personal and even more powerful. For an entire list go to the pen-based interface section of the bibliography.
There are, in my mind, two main reasons to focus on digitizer pens: expressivity and flow.
With modern digitizer systems the pen is not just a cursor with a binary "click" like a mouse, but also offers pressure levels. When implemented, this allows for finer expression from the user and is a richer datasource for developers than traditional yes/no mouse clicks. Unfortunately, the current digitizer pen systems do not have web APIs, meaning that such experiments in utilizing pressure for interface are bound to locally compiled software.
More importantly though, writing with a pen on paper, a user can experience a state of flow: when their intention is channeled through their hand and into forms of any kind that they need to represent what they desire. Paper has no "tool modes" to select. This is the key to staying in flow: no switching modes. With a pen you are using one tool to express your intention and nuance comes from physical movements of your hands relative to other physical movements that have been recorded as ink. Using a pen on paper satisfies Norman's principles easily.
Paper is obviously limited in its scope though, so the question becomes: can digitzer pen systems be utilized in a similar slow-enducing single mode?
My experience using applications like Adobe Illustrator and Microsoft OneNote is a strange one. Both of these software systems, and indeed most software for desktop computers, were designed with the mouse and mode-switching in mind. So when I want different color inks in OneNote I need to switch modes to another pen. When I want to draw shapes in Illustrator then draw a path I need to be in a specific mode for each one. This ability to have essentially an infinitly-modal pen is very powerful, it is in keeping with the metamedium possibilities of computers and it makes sketching out and implementing graphical ideas much more visceral and intuitive.
Sketches do not need to be bitmaps, they can be comprised of points along vector paths. This allows every point and the delta between the points to already be encoded into a regcogntion system.
The SILK (Sketching Interfaces Like Krazy) system is one of the most fascinating, expecially when compared with modern day software design tools, such as Google's Pixate and Adobe's Experience Design. These systems are set up primarily to create applications in the style that we are now familiar of. They are machinery for creating 'killer apps'. This is helpful if you seek to create apps for consumption, yet it carries a big risk of funneling designers into a trap: if you have a tool that only makes killer apps you are like a carpenter with only a hammer, the world will look a certain way to you and you will probably end up banging away on things when you should have done something else.
That's why SILK is so fascinating, it does not start you in pre-designed templates but instead allows and encourages you to literally draw from your imagination. This creates a tool that is much more flexible and expressive.
What I think we need is a prototyping environment where we can bring in everything that has so far been utlized in interactive computing from the entire WIMP paradigm to multitouch gestures and then set it all free into a free-sketching environment. A combined environment that featured UML diagram recognition, database sketching and interface sketching would allow rapid prototyping of a much larger set of possible interaction paradigms. Does every menu have to go on the upper left side of a mobile app? What would happen if you drew a whole new and crazy kind of interface? Like so many things in design, you often cannot know until you try. I am not arguing that the modern design aesthetic and practices of apps and websites is wrong, I am merely cautioning that we could get stuck in it if our tools are too rigid.
There is a fine line here between automating design and augmenting human designers. In a fascinating talk by Tim Ewald titled "Clojure: Programming with Hand Tools" he describes the change from hand tool-based woodworking to today's modern machine woodworking. Of particular min-blowing interest to me is that in the hand tool days blueprints did not have instructions like we know them today, instead they were all based on proportion and geometric rules.
Ewald seems to think he is stretching the analogy of woodworking to programming, but I think it is not being explored enough. I want to experiment with how to literally program with actual hand tools. The visceral artistic connection achieved by working with ones hands could usher in a generation of programmers who are more connected to their work. We could see a shift from pyramids of code to cathedrals of code as Alan Kay has long urged us to move toward.
The diagram showing the essential aspects of their theory.
What is human thought anyway? What are we getting up to when we imagine things or reason through problems?
Gilles Fauconnier and Mark Turner attemped to provide a framework for thinking about thinking with their illustration of "conceptual blending."
They illustrate this idea with a riddle and diagram out the steps.
A Buddhist Monk begins at dawn one day walking up a mountain, reaches the top at sunset, meditates at the top for several days until one dawn when he begins to walk back to the foot of the mountain, which he reaches at sunset. Make no assumptions about his starting or stop- ping or about his pace during the trips. Riddle: Is there a place on the path that the monk occupies at the same hour of the day on the two separate journeys?
In Fauconnier and Turner's view, this riddle is solved through the creation of an "integration network" and the manipulation of "cross-space mappings". The whole sequence that they present is:
This conception is compelling to me because it is presented immediately with a visual argument. Even if this theory does not completely describe human cognition, it offers a very concrete way to begin building systems that could let users manipulate and blend symbols which stand for "mental spaces" and see if they can get emergent patterns.
Full disclosure I do not know enough about programming to truly articulate the whole dynaPlane interaction paradigm, let along execute it. But I think that a project like this will be the key to me learning programming. What better way to learn math than to try and build a version of mathland?
For me, there are three stages to this dynaPlane vision.
To achieve a version of dynaPlane with no modes would mean that the software system would effectively have to guess what the user is planning on doing next, which is a daunting problem set.
The vision is still very fuzzy in my mind because I have not yet been able to build anything to actually get my hands on. I think this will change. I have just discovered an IDE called Light Table which was written in the functional language Clojure. Light Table allows you to string together functions outside of files and inject your own code onto live web pages and many other scripts and markup languages. The entire idea of functional programming is something that I have just discovered: instead of functions causing side effects of assigning variables state, the state of the entire program is one value and all objects and variables are referenced, never changed directly only changed inside of functions. At least as far as I've understood. This seems like a powerful conception though, since I can imagine a moment when, if it was in a functional language, an entire windowed operating system could be used as a variable for another function!
All this will surely update as I learn more. I feel as though a weight is being lifted from my shoulders the more I explore the ideas here because even if I can not make these dreams a reality, at least I am showing myself that I can learn to understand programming. I am gaining the literacy of the cybernetic world, and I hope to help pass it on.