MIT application statement of purpose

The dream of self-augmentation in the future

I do not foresee a future where we build Turing test-passing machines, though we could (see my informal essay on this topic at the above URL). Rather, we will house the machines in our bodies, building on top of the intricate, robust machinery that already exists in our brains, just as evolution has in the past. Self-augmentation is the ultimate goal, but plug-and-play infra-red vision and hot-swappable wetware memories remain a distant dream.

For now, we have to work with the machines that we have. The challenge is to incorporate some of the robustness, flexibility, and richness and variety of representation that we see in ourselves. Whether we’re talking about a robot arm trying to tell objects apart visually, a mobile phone deciding whether or not to ring out loud, or an adaptive user interface (UI) that starts to infer patterns of use and build a user model, the same problems of representation, learning and inference are central. These are all attempts to build systems to perform difficult tasks in a complex, dynamic environment, by finding and exploiting regularities in an often noisy, high-dimensional space, ideally with as little hand-coding/hard-wiring as possible. In every such case, designers can benefit from lessons learnt from biological systems, cognitive psychology, careful experimentation, and the occasional injection of philosophical rigour. For all these reasons, I hope my inter-disciplinary background might provide a valuable perspective (see below).

The dream of self-augmentation in the present

My laptop is my external brain. I depend on it as a memory aid, repository, toolkit, newspaper, way of staying in touch with my friends (and occasionally footstool). We can’t build this functionality into our body-based brains yet. But we can improve the interface between internal and external brains, and we can offload more of the workload. This is the dream of self-augmentation in the present.

User interfaces could be better than they are now. The way that information is organized rarely reflects what it’s about, and so finding things is difficult and slow. I find myself continually performing repetitive tasks, worrying about whether my most important files are backed up, and being distracted by trivialities and interruptions. My external laptop brain is actually adding to the workload of my body-based brain, and unnecessarily confusing the interface between them. What can we do about this?

We can see this problem in terms of the short- and long-term vision, and the types of problems that we need to address in each. In the short term, we can try and tackle these issues with better design and a better metaphor for the user interface than the desktop and paper files and folders (for example, I find David Gelernter’s work on Lifestreams an intriguing alternative). We can build a richer set of user preferences, and hand-code different behaviours correspondingly. But if we are to make real progress in this area, we need to focus on the lessons to be learnt from context aware and common sense research. User interfaces have contexts, just like the real world. These could be broad ones, like ‘hard at work’, ‘perusing the news’ and ‘playing games’, or they could be much more fine-grained, such as ‘working on the project proposal for my MIT application’, ‘learning how to use a function in a new toolkit’ or ‘looking for romance on instant messenger’. In each case, the system needs to take into account all the contextual factors, including the physical environment, social context, user’s emotional/physiological state and level of expertise, and the task domain and goal. Only then can we apply common sense assertions to try and make sense of what is happening, what the user will and won’t want to happen, what to prioritize and eventually, how to best organize their information.

In the longer-term, we want to try and get our system to learn about the user over time, and adapt itself accordingly. It may start with some preconceived notions of how most users operate in a variety of standard contexts, but every person’s needs, skills and habits are different, and change over time. This is a problem of self-organization, of building a rich, high-dimensional representation of the user’s interaction with the system, and eventually generates its own common sense assertions out of the regularities and structure that emerges.

[snip]

---

Genuine context-awareness is an AI-complete goal. To fully appreciate how people take all the nuances of a given situation into account demands the same human-level understanding as passing the Turing test. But that doesn’t mean that we can’t make headway towards systems that reliably respond in a more intelligent fashion than they do now – this rests on the assumption that we can map the incredibly high-dimensional space of the real world onto a much lower-dimensional space (a suite of sensors, say), and still retain most of the information we need to assess context.

User interfaces provide a constrained environment about which we have almost complete knowledge, while being complex and nuanced enough to provide a theoretically interesting test-bed. The factors that influence how people use and respond to their computers span the spectrum of contextual dimensions, such as physical and social environment, emotional state, level of expertise, nature of task and the computer system itself.

Possible applications

I can foresee a huge range of possible applications. I have listed a few here, to illustrate what I have in mind.

Keeping things as simple, or as complicated, as the user desires/requires, e.g.

Hiding and displaying advanced features.

Providing tailored wizards and help where necessary.

Building up a picture of associated tasks/activities, and patterns of use, so that the system can:

Relate and organize documents, emails, webpages and contacts, based on temporal coincidence, statistical correlations in text, storage location etc.

Make finding specific or generally relevant information easier and faster.

Predict which applications and files the user is likely to need, and pre-load them in the background.

Automatically alter instant messenger (IM) availability status to reflect whether the user is hard at work, perusing the news, playing games or responding to email – furthermore, this availability is probably different for wives, bosses and friends.

Tenets

The following three tenets should be permanently at the back of the designer’s mind.

Adaptivity

The system should learn for itself. This functionality should not require the user to state their preferences in advance or explain their patterns of usage explicitly – indeed, users find this very difficult to do in the abstract.

Transparency

Signal all adaptive decisions that are made (unobtrusively).

Allow the user complete control over the level of adaptivity.

When the system doesn’t know or is confused about what the user wants, it should say, or make a neutral decision.

Err on the side of caution

The MS Word paperclip highlights the most important lesson in context aware research – no adaptivity is hugely preferable to intrusive, error-prone interference. This threshold of acceptable error is very low.

Problems

Despite being much cleaner than the real world, a personal computer is still a hugely complex environment, and presents the three biggest problems common to almost all context aware projects.

Limited knowledge

In the worst case scenario, the designer may not have complete access to all of the OS’s underlying processes and messages. Even then, there are a huge number of valuable clues about the outside world that we rely on that aren’t available to our PC, e.g. that I’m on a train, that the phone is ringing, or that I’m irritated because I’ve split coffee on my tie.

Context switching

The system has to have some means of dealing with different users sharing the same computer without logging in and out. The system has to be able to deal with abrupt changes in task, and with juggling more than one context at once – I may have multiple windows and applications open, and jump between them as ideas occur to me, in response to phone calls or while I’m waiting for something to happen.

People

People are complicated and unpredictable, even to each other, and especially to computers. They have personal agendas, social agendas, habits, conventions, irrationalities and whims. They are the highest-dimensional space of all, and can only be imperfectly mapped onto a lower-dimensional space.

Techniques and resources

To this end, we can utilize various techniques and resources to improve the user experience.

Hard-wired rules

Often, it’s simply easier to hand-code certain rules and heuristics, and there will always be certain incontrovertible rules that underpin the rest of a system. Unfortunately, this approach doesn’t self-organize, tends to be brittle, and is labor-intensive.

User-specified preferences

There need to be some mechanisms to allow the user to view, edit and add to the knowledge and rules the system is employing.

Common sense

This is one area where the Media Lab is particularly future-looking. It is absolutely critical that an adaptive UI employs some sort of common sense in the way it works, as a means of regulating and shaping the sort of rules it comes up with, and because common sense knowledge is so distilled. The sort of common sense nuggets I have in mind might go along these lines:

·        Files in 'My Documents' are probably data.

·        Data is important.

·        Files with ‘.bak’ or containing 'backup' in their name are probably duplicates.

·        If a folder is contained inside another folder, which is contained inside a further folder, the base folder contains them both.

·        Big files take longer to move than small files.

·        Usually, the things on the desktop are useful or recent.

·        If I’m typing in Microsoft Word, I’m probably working.

·        If I’m working, I probably don’t want to chat with friends on IM.

I’m not yet sure how best to go about harvesting, encoding or using knowledge like this, or in fact whether it would prove too sparse to do useful inference upon. Presumably, many of the same issues that polarize Cyc and Open Mind in the wider domain of real world common sense would crop up.

I was pleased to find that there is already a small amount of this type of information in Open Mind, e.g.:

·        ‘A file that gets deleted from your Windows desktop goes into the computer recycling bin’

·        ‘The active window is the foremost window on the desktop’.

However, the assertions would probably have to be hand-filtered, to avoid confusing statements from a different context like ‘A file is a folder of documents’, and many more would be needed.

Self-organization

One extreme way of self-organizing would be to try to encode every action, process and message in the system and use statistical, connectionist or machine learning techniques to see regularities and correlations between mouse movements, documents being opened, time of day, active windows, present network connections etc. Each of these modalities of information can be seen as a sensor, providing some information about the current context, that may or may not be relevant, but together, we might hope to see some useful structure emerge.

Much of the common sense information described above should be derivable in this way. We can imagine how a system might eventually learn to prioritise information according to the number of backups we make, how often it’s accessed, whether it’s referred to in emails marked ‘urgent’ etc. In the longer-term, self-organised common sense knowledge may prove more powerful, but aren’t yet in a position to build such systems.