ML Intelligence

I would like to turn for a moment to the discussion of duality. Commonly when one speaks of a duality in reference to the mind body problem they are speaking of the existence of a physical substance and a non-physical mind substance. However in my recent contemplations I have narrowed the search for the dualistic properties of the mind down to the interaction between a physical substance, matter, and a non-substance, which is simple the substance of meaning and logic. It seems to me that these two things, while not commonly set in contrast to each other may play the roles of the physical substance and mind substance that are postulated in the dualistic approach.

What I mean in regards to matter is simply the material of which physicists are investigators of. Matter here is simply the collection of particles and energy in the universe. Particles which jostle around, perhaps clump together, but have no meaning or purpose. That is from a truly physical perspective you are just a mass of molecules, most of them water molecules jiggling around in a large mass. This mass of particles may have physical properties such as mass, and velocity, but without a definition of a boundary between the particles that form you and those that are not, such measurements are not feasible.

Now, in addition to your physical substance you are composed also of meaning. This meaning is what your identity is composed of. Without meaning, the mass of particles that compose your top half may as well be part of a different subject than those that form your bottom half. Meaning has a substance all it's own. The organization of this substance is studied by ontologists. It is the material of mathematics. It is what this blog post is made out of. Ideas, meanings, definitions all belong to this second class of substance.

It may be difficult to separate these to substances and perceive them as being altogether different. This is because our minds are on the threshold between these two worlds. One world composed of only substance, and the other of only meaning. These two universes are molded together in that one can be mapped onto the other. Matter can take on forms, spheres, people. Whereas some parts of the physical world resist definition. Where is the boundary of the universe and what lies beyond it, for example? Whereas there are some aspects of meaning that cannot be represented by matter, like a perfect circle. So where these worlds do unite and in as much as they are capable of overlapping is where we as conscious beings exist.

A neural network is a mathematical model that is inspired by the function of neurons in brains. These networks of virtual neurons have been successfully used to solve a large set of computational challenges. They have been used in pattern classification as well as for prediction and forecasting. Despite the success in certain applications, neural networks have not achieved the results that were anticipated decades ago in the excitement that prevailed when they were just beginning to show results. But as the years go by it seems that progress has not moved as fast as expected. Many researchers have deserted the neural network approach for a more computationally friendly and mathematically elegant approach, namely support vector machines.

So what happened? If biological neurons have the ability to solve some of the most complex problems, reasoning and vision for example, why can’t this success be duplicated with virtual neurons? Currently research efforts towards this end are focused on brain mapping and imaging. The goal is to obtain a massively high resolution map of the physical, chemical and electrical properties the brain. Presumably this data will be the key to unlock the secrets of the mind and will lead to advances in neural computation approaches.  

Perhaps we have all been looking in the wrong place. Perhaps the real problem is not lack of information but rather false assumptions that have been made about the purpose and role that neurons play in biological systems. Is it possible that machine learning researches simply don’t understand the function of neurons in our brains? Let me use an analogy. It would be like trying to reveal the source code of Microsoft Office by placing your hard disk under a microscope. Trying to decode the secrets of a computer program would be very difficult without knowledge of a computer language. So maybe there is some underlying misunderstanding of what the brain is actually doing and which functions it is performing. The distinction between the physical and the logical needs to be better understood.

Now the particular assumption that I am referring to is one that is made as a result of mathematical thinking. The assumption is about the function of artificial neural networks as they have been adopted by computer scientist. An example of neural network application is to take as input some set of values that represent the properties of an object, perform some computation and to output a number that correctly classifies the input. This idea that neurons are just fancy calculators used to compute the solution to a given problem is a false assumption. In reality neurons do not simply crunch information to produce the knowledge needed to take some action. In reality neurons go beyond this by complete the cycle by actually taking action.

The missing half of the neural network problem is action. Your neurons not only can identify that your hand is on a hot stove, it goes ahead and moves your hand away, and that is even before being processed by the cortex. Behavior is just as fundamental to biological neurons as observation.  Ever since the neuron was invented more than 500 million years ago according to evolutionary theory they have been used to enable movement and action. Almost all networks of neurons in organisms have a cycle that binds observation to behavior in a feedback loop.

This looping process incorporates a space in which a body can sense and take action within. But despite the fact that it would be hard to find an example of a real neural network that does not incorporate this process, it has been completely ignored by computer scientists in favor of a more utilitarian model that simply makes information available to a foreign observer. Perhaps developing a system to do whatever is wants to do does not seem to be the most useful thing to do. How can anyone get ahead by developing software that doesn’t provide useful information but rather it interacts with the universe and just does whatever it wants? The assumption that a neural network should be useful solely to provide knowledge is misleading. What is needed is to research the emergent properties of the behavior, observation cycle that is so fundamental to life.

Focusing on the virtue of elegant mathematical models will surely mislead any inquirer of the nature of the mind. Most of the modern mathematical language is directly sourced to Sir Isaac Newton and his theories of the physical world. Simply because a logical relationship or a functions of variables can be written with cool looking Greek symbols and integrals does not mean it is the most usefull representation. A universally held assumption, held by those in the machine learning community, is that object labels or classifications are functions of physical quantities, namely vectors. And that they behave like physical phenomenon. But the classification of something is surely in the eye of the beholder and therefore it belongs to the domain of the mind.

Ever since Galileo and his telescope overturned the spiritual worldview and gave birth to the scientific or physicalistic perspective, scientists have generally believed that all things reduce to the physical. Only a small group of scientists, mostly philosophical or religious types, prescribed to the notion of a fundamental dualism of mind and body.

Galileo Galilei

The idea that there is a substance that is altogether different from anything that can be found in the physical world, that there exists a mind separate from the body was expressed well by Rene Descartes in his Meditations on First Philosophy:

Moreover, I find in myself diverse faculties of thinking that have each their special mode: for example, I find I possess the faculties of imagining and perceiving, without which I can indeed clearly and distinctly conceive myself as entire, but I cannot reciprocally conceive them without conceiving myself, that is to say, without an intelligent substance in which they reside, for [in the notion we have of them, or to use the terms of the schools] in their formal concept, they comprise some sort of intellection; whence I perceive that they are distinct from myself as modes are from things. I remark likewise certain other faculties, as the power of changing place, of assuming diverse figures, and the like, that cannot be conceived and cannot therefore exist, any more than the preceding, apart from a substance in which they inhere. It is very evident, however, that these faculties, if they really exist, must belong to some corporeal or extended substance, since in their clear and distinct concept there is contained some sort of extension, but no intellection at all. Further, I cannot doubt but that there is in me a certain passive faculty of perception, that is, of receiving and taking knowledge of the ideas of sensible things; but this would be useless to me, if there did not also exist in me, or in some other thing, another active faculty capable of forming and producing those ideas. But this active faculty cannot be in me [in as far as I am but a thinking thing], seeing that it does not presuppose thought, and also that those ideas are frequently produced in my mind without my contributing to it in any way, and even frequently contrary to my will. This faculty must therefore exist in some substance different from me, in which all the objective reality of the ideas that are produced by this faculty is contained formally or eminently, as I before remarked; and this substance is either a body, that is to say, a corporeal nature in which is contained formally [and in effect] all that is objectively [and by representation] in those ideas; or it is God himself, or some other creature, of a rank superior to body, in which the same is contained eminently. But as God is no deceiver, it is manifest that he does not of himself and immediately communicate those ideas to me, nor even by the intervention of any creature in which their objective reality is not formally, but only eminently, contained. For as he has given me no faculty whereby I can discover this to be the case, but, on the contrary, a very strong inclination to believe that those ideas arise from corporeal objects, I do not see how he could be vindicated from the charge of deceit, if in truth they proceeded from any other source, or were produced by other causes than corporeal things: and accordingly it must be concluded, that corporeal objects exist. Nevertheless, they are not perhaps exactly such as we perceive by the senses, for their comprehension by the senses is, in many instances, very obscure and confused; but it is at least necessary to admit that all which I clearly and distinctly conceive as in them, that is, generally speaking all that is comprehended in the object of speculative geometry, really exists external to me. 

The above reasoning does not appeal to our modern day machine learning scientists. The distinction between the physical laws of Newton and the functions of the mind needs to be drawn before progress will be made in the field of machine learning.

René Descartes

Drawing from Meditations on First Philosophy

On Intelligence - Jeff Hawkins

His focus on studying the function of the brain brings an outsiders perspective to machine learning. He has developed a new machine learning framework called Hierarchal Temporal Models. What makes this approach different is that it is based on discoveries made about the organization of the neo cortex, and specifically how models of patterns are stored hierarchaly. His company, Numenta, Inc. currently owns the patents for this technology. According the Numenta website the name Hierarchical Temporal Memory is defined as follows:

Hierarchical -- HTMs are organized as a tree-shaped hierarchy of nodes. Each node implements a learning and memory function, that is, it encapsulates an algorithm. Lower-level nodes receive large amounts of input and send processed input up to the next level. In that way, the HTM Network abstracts the information as it is passed up the hierarchy.

Temporal -- During training, the HTM application must be presented with objects as they change over time. For example, during training of the Pictures application, the images are presented first top to bottom, then left to right as if the image were moving over time. Note that the temporal element is critical: The algorithm has been written to expect input that changes gradually over time.

Memory -- An HTM application works in two stages, which can be thought of as training memory and using memory. During training, the HTM Network learns to recognize patterns in the input it receives. Each level in the hierarchy is trained separately. In the fully trained HTM Network, each level in the hierarchy knows -- has in memory -- all the objects in its world. During inference, when the HTM Network is presented with new objects, it can determine the likelihood that an object is one of the already known objects.

What is great about Mr. Hawkins style and method is that he is more focused on practical functions such as the actual function of the brain and less on abstract mathematical concepts.

Today I have compiled a market research report of publicly traded companies here in the United Sates that have significant intellectual property in machine learning. Of course there are additional firms that have made advances in machine learning. Some of the machine learning companies that did not make it into this report because they are either private companies such as Numenta, Inc. or foreign such as NEC Corp. will be discussed in later posts.

Also, I have installed a stock ticker on the right to track the progress of these machine learning innovators in the marketplace. Each machine learning firm detailed in this report has received five or more patents for machine learning technologies. Some of the relevant patents are listed. Also an excerpt from the companies website pertaining to machine learning technology has been included as well as a direct link to the referenced material.

What makes machine learning such an exciting field is not only its philosophical implications towards the mind and intelligence but also its widespread adoption into much of today’s cutting edge technology. You may be surprised to discover just how much new technology is driven by machine learning. Most new digital cameras incorporate face recognition, automobile engines optimize fuel efficiency with machine learning algorithims, banks use it to monitor your credit card spending for fraud. But one of the most exciting, and lest well understood, applications  is at the core of Google’s search technology.

Google’s methods have been a well kept secret. If you where to ask Google how their search engine works they will refer you to their Pigeon Rank technology website which satirically explains how pigeons are used to rank web pages. Google is not about to make its intellectual property public knowledge. If it did Microsoft would surely have a better search engine.

But, the nature of Google search technology is not completely hidden. Patent applications as well as job posting,  research publications and professional conferences reveal just how important machine learning is to Google.

The very first United States Patent Issued to Google was a “Large scale machine learning systems and methods”, patent number 7,222,127 originally filed on December 15, 2003. The language of the patent is very intentionally vague, referring not to web sites, or to web pages, but rather only  to “nodes.“

What exactly is this invention? Well according to the patent it is:

 A system for generating a model, comprising: a plurality of nodes, at least one of the nodes being configured to: receive candidate conditions, generate statistics associated with at least a first one of the candidate conditions, send the generated statistics to at least one other one of the nodes, receive statistics regarding at least a second one of the candidate conditions from other ones of the nodes, form a rule based, at least in part, on the second candidate condition and the received statistics, add the rule to the model, and outputting the rule to at least one of the nodes.  

Looking past the obfuscation, it is a method for modeling documents such as web pages for the purpose of classifying them. It builds a model of a network by indentifying conditional relationships between pairs of nodes based on special, not very well defined, conditons. It is intentionally applied so broadly, using terms like node, that it could apply to jsut about anything.

The patent is summurized as follows:  

Systems and methods consistent with the principles of the invention may apply machine learning to large data sets, such as data sets including over one hundred thousand features and/or one million instances. The systems and methods may be capable of processing a large data set in a reasonable amount of time to generate a classification model.

Different models may be generated for use in different contexts. For example, in an exemplary e-mail context, a model may be generated to classify e-mail as either spam or normal (non-spam) e-mail. In an exemplary advertisement context, a model may be generated to estimate the probability that a user will click on a particular advertisement. In an exemplary document ranking context, a model may be generated in connection with a search to estimate the probability that a user will find a particular search result relevant. Other models may be generated in other contexts where a large number of data items exist as training data to train the model.

A more serious diagram. Figure 1 from Google Patent 7,222,127

Figure 5 from Google Patent number 7,222,127

This patent is only the first one Google received. Searching the USPTO.gov website reveals dozens of additional patent applications as well as issued patents that relate to machine learning techniques. 

Allot of money has and will continue to be invested into Machine Learning technology. Learning algorithims are at the heart of todays most exciting research so this is a great time to get involved. 

In plain English, machine learning is the field where researches program computers to recognize patterns, the kind of patterns our brains recognize with ease. How do you recognize your friend’s voice over the telephone, or how can you tell the difference between a cat and a dog? These tasks are trivial for you and I, but computers struggle with even the most simple pattern recognition tasks, despite decades of research.

You can classify these images, but can a computer?

While no one knows precisely how our brains can identify the subjects of the above photos, some smart scientists have been working on the problem of pattern classification from a mathematical perspective. They have created a standard by which a classification problem can be broken down into a system of variables that can be analyzed using statistical methods. Let’s see how this works. Let’s take the example of how to recognize if a photograph contains an image of a dog or a cat. Let us ignore for now the possibility that the photograph is of neither, or both.

The first step in a machine learning application is what is called feature extraction. An example of a simple feature could be the length or height of the subject. Also, more complex features could be the shape of the head, the posture, etc. Once the features of the subject in question have been identified and given a value, for example 18 inches tall and 25 inches in length, the machine learning algorithm will compare these values statistically with the values of known subjects. One can visualize this process on a table where the horizontal axis represents length and the vertical axis represents height. Dogs are known to be bigger than cats for example, so generally a cat will be represented by a dot at the lower left, and a dog at the upper right. In theory, by plotting the features this way, there should be two distinct regions separated by the classification boundary. The goal of any good machine learning algorithm is to find the optimal boundary. The problem with this method is clear, obviously not all dogs are taller and longer than all cats. And what about a catdog? We can recognize one without confusion, but any machine learning algorithm would struggle with such a classification.

Classification plot of length and height of sample dogs and cats.

There is clearly a lot of room in machine learning for innovation. Keep in mind also that the example given here is only a simplistic example. But, generally the problem of feature extraction is one that is at the core of machine learning.

This is an excellent online resource for an introduction to Machine Learning. In a series of 20 lectures, Professor Andrew Ng introduces his students to the rigorous mathematics of Machine Learning. His focus on the purely mathematical and whimsical mockery of graphical models will surely confound those who dislike math. But I recommend these lectures to anyone who is a serious student of machine learning because it will help to build a strong foundation of mathematical principals needed to understand the perspective of the machine learning community. Also check out the course hand outs. These are nearly as detailed and information packed as a machine learning text book, and they are free.

The course hand out website is:
http://www.stanford.edu/class/cs229/materials.html

The playlist website is:
http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599