Skip to content

Mirroring, Mindreading and Simulation of the (Java) Virtual Machine

A widely recognized source of fruitful innovation is the cross pollination of ideas from one field to another. So it stands to reason that some innovations, not inspired by exchanges of ideas across domains, may already be present in other fields. Finding such cases of parallelism can be extremely useful to the inventor by way of analogy in communication and validation prior to greater acceptance and agreement. It may also hint at future evolutions and applications of the innovation.

Theory of Mind
The Theory of Mind (ToM) is a branch in cognitive science that looks to define a framework that describes the underlying process and mechanism to how we ascribe mental states to other persons and how we use such states to explain and predict the actions of those other persons. It investigates our ability to mindread and/or mentalize. There are two established contrasting theories within this branch, which actually takes its name from the earliest established theory  - theory of mind (theory-theory). From the theory-theory perspective our ability to read minds is based on somewhat detached theoretical process that is innate. This ability rests upon knowledge of a theory, although we’re not actually aware of the laws which comprise the theory. The knowledge that this theory represents holds much the same structure as a scientific theory in its acquisition and storage, and is used in much the same manner in a formulation of a question, hypothesis and prediction, and then its revision in light of new evidence.

“Mind reading is the ability to attribute goals and motivations to others based on the observation of their behavior.”

Simulation Theory
The second theory and the focus of this article is the simulation theory. This theory suggests that we understand others through our own mental apparatus. We form predictions and explanations of someone by putting ourselves in the shoes of another person and simulating them. The simulation generates pretend mental states, which are expected to correspond to those of the target. The simulation then feeds the pretend states into a suitable cognitive mechanism that is taken “offline” in that the motor control system (actions) is disengaged. Finally the simulation projects states onto the target.

“The simulation theory holds that humans anticipate and make sense of the behavior of others by activating mental processes that, if carried into action, would produce similar behavior. This includes intentional behavior as well as the expression of emotions.”

imitation.mirror.simulation.1

Mirror Neurons
Most simulation theorists propose that a mirroring system, consisting of mirror neurons, underlies the simulation process. The mirroring system provides a mechanism to couple perception with action and to some extent the goal of the action. When the mirror neurons are activated in observation of some target, the firing activity is matched with the result being a representation of the intention (goal). A mirror neuron fires both when an act is performed and when the same act is performed by another. The passive activation of a mirror neuron reflects the very action that is being observed. It is this pattern of activations that is similar to the actual response from the self execution of a similar action. Effectively there is a correlation of the sensory feedback generated by the action and the observation of the equivalent action in terms of goal.

“Mirroring is formed by activations within a neural cortical system when a person experiences an event endogenously or when the person observes (perceives) an action by another person that correlates to the event in some manner.”

imitation.mirror.simulation.2

The mirrored simulation can be used to retrodict as well as predict mental states, allowing us to work backwards and draw an inference from the observed action to a hypothesized goal state as well as generate explanations of the target’s behavior. A not entirely accurate analogy in the context of our domain of expertise would be a crime scene profiler in which evidence and signs are used to reconstruct the internal states by way of actions (infused symbolically).

On Dreams: The threat simulation theory, which we sometimes jokily refer to as the thread simulation theory, claims that dreaming evolved as an offline simulation to repeatedly experience life threatening (threading) events as a means to increase survival rates in priming response behavior.  

Machine Minds
Lets now turn our attention to how we design machines to observe, monitor and hopefully manage other machines. The traditional approach to application monitoring is to have a separate monitoring process, an agent, routinely query (pull/ping) the internal state of another process, an application, delivering a service.

imitation.mirror.simulation.3

The diagram above calls out the fact that the monitoring agent is blind to the actual internal software execution behavior that generates (or represents) the states which are returned as results to queries. For the agent to function it must already have some knowledge of what to ask and an understanding of the answers. Importantly neither process has any degree of “self-awareness” – reflection on its own execution behaviour. Typically the responses returned to the agent are recorded in an external storage system which is then data mined for reporting and analytical purposes. The link between perception-action-state(goal) is completely lost here. It is impossible to reconstruct and simulate the behaviour only the answers to questions submitted during observation. We can only understand what we had already learnt.

What if instead we modeled our approach to monitoring based on the simulation theory. First we would need to be able to observe the actual software execution behavior – perceive the motor action. Instead of waiting for an agent to pull metrics the observed application emits activity metering events outwards. We don’t want to overload both the channel and its subscribers so an adaptive learning mechanism is employed locally to dynamically limit the code instrumentation and activity metering (measurement) to those aspects that represent the essence of the execution behavior. We call this Intelligent Activity Metering (IAM).

imitation.mirror.simulation.4

Now that we have information (light) emitting from the application, hitting the sensory system in our mirroring application, we need to process the events, extract the perception(-action) data and then use this to drive a simulation of what behavior is occurring in the observed application. If you recall the mirroring mechanism within the brain shares a similar circuitry to that of the observed brain. In the case of Simz, our unintentional realization of the simulation theory, that shared circuitry is in fact the metering engine. The calls into the metering engine that occur within the observed application are replayed, nearly identically, within the mirroring application. What Simz does is transform the event data in the metering feed into mirrored execution constructs (threads), resources (meter readings), and invocations (API calls) within the observing application. It transforms and translates what is perceived into simulated (behavioral) actions. The simulating application effectively imitates the other observed application and in turn it becomes the application to other observers of itself.

imitation.mirror.simulation.5

But how do we go from action to goal and intent especially as there can be many slight variations in the event stream for a particular action such as the processing of a request? We need a form of complex event processing (CEP) to perform a matching within the context of a threads execution. This is what Signals offers on top of our metering engine, though it can also operate independent to the metering engine and its simulation if need be. Rules attached at various nested contextual boundaries (organizational cell structures) turn low level signal firings (neural activations) into higher level stimulus that brings us closer to the actual intent or internal state of the observed application even when the application itself is not self aware (sensory deficient). The mirroring application takes on the role of the trained (and experienced) mind.

imitation.mirror.simulation.6

What of electric sheep? Can the machine use dreams in the same way we train ourselves to better predict and recognize, as well as avoid, dangerous situations and behavioral patterns? Yes. OpenCore allows metering to be recorded online and then played back (simulated) offline. In the playback we can even feed this recording to an external observer such as Simz. We can go as far as have Simz receive metering feed from both online and offline simulations. Since recordings can be played back repeatedly we can use this to influence and train the simulation and signals processing of online feeds – a form of supplementary on the job training or reconditioning.

imitation.mirror.simulation.7

Even in an online simulation mode the mirroring application can process neural-like stimuli originating from multiple observed target (machines). In doing so its adaptive learning of the perception-action-goal chains can be more representative of the overall machine population (collective).

imitation.mirror.simulation.8

Likewise we can have multiple mirroring applications each with slightly different operational settings creating a form of crowd sourcing in the understanding of what is observed and how it should be interpreted.

imitation.mirror.simulation.8b

Whilst the mirroring application does not actually perform the observed behavior, but instead processes the same stream of response stimuli, it can take on a far more important and substantive role in becoming the nervous system for a “borg” like collective of virtual machines, relaying signals back to application which in turn trigger local adaptive mechanisms leading to new behavioral patterns that activate entire new signals to be relayed back and so on. The observer creates its own reality (or universe) in a participatory role as the simulation folds itself on that what it observes which is possible because of a shared circuitry in the use of both the metering and signal engines. The simulation is a true collective in the sense that it mirrors the behavior across the entire application population.

imitation.mirror.simulation.9

On Big Data Analytics: The current approach to Big Data resembles the theory-theory approach to ToM. Far to much emphasis is placed on data collection and storage, and the formulation of (offline) questions impacted by laws not fully understood. There is no associative link between perception, action and intention. Not to mention the time delay in this process is not practical in achieving real understanding that has an incredibly short life span within some interaction context. A possible better approach, and one that nature has already deemed optimal in its evolution of man and his mind, involves real-time simulation coupled with stimuli signaling that adaptively generates or tests new actionable responses. Since human interaction is in a way reflected in a machines execution behavior we can extend the simulation to users much like in online virtual worlds.

Further Reading
Changing Space and Time in Software Execution – The Future is Simulated
Rewriting Application Performance Monitoring Histories with Record & Playback
Simulation and Time Synchronization of Application Memories