A New Computing Paradigm

The AI revolution has captured people’s imagination, but it also has brought to the forefront various issues with the current AI stack. Hallucinations and overconfidence in large language models are very active fields of research and have highlighted the need for reliable, adaptive, and auditable AI. However, principled probabilistic reasoning - which is the basis for making AI reliable - is computationally expensive. This emphasizes the need for probabilistic hardware accelerators. 

Indeed, the need for novel computing hardware has been emphasized by Geoffrey Hinton, who noted the mismatch between modern AI algorithms and current digital hardware. Hinton proposed a future where hardware and software are inseparable, and where the hardware is variable, stochastic, and “mortal”.

Thermodynamic AI represents a step towards this vision. It is a new computing architecture where the fundamental building blocks are inherently stochastic. An example of this is shown in the circuit diagram below, involving a resistor, capacitor, and voltage noise source, such that the voltage at point 1 evolves stochastically over time.

Our R&D team at Normal Computing detailed this physics-based computing paradigm in February in this arxiv paper, and we recently extended the application space of Thermodynamic Hardware to linear algebra primitives. (See this talk for an overview.) Now, we are giving everyone the opportunity to explore these theoretical ideas in a live Playground.

The Thermo Playground

Here we present our “Thermo Playground”, which is the subject of this blog post. The goal of the Thermo Playground is to give the user a feel of what it would be like to interact with the first iteration of Thermodynamic computing hardware.  Try out this playground, and enjoy!

We invite the reader to join us on a journey into the future of computing, where the hardware is inherently uncertain, probabilistic, and physics-based. The Thermo Playground gives a glimpse into this exciting future. It will give you a feel for the various hardware dials and free parameters in Thermodynamic AI Hardware, and hopefully it will also show you how to start thinking thermodynamically, with the end goal of writing software and algorithms for this hardware.

In addition to illustrating the Thermo Playground, this blog post will highlight a key use-case of Thermodynamic Hardware, namely, for enabling large-scale methods for reliable AI.

Unlocking reliable AI

Imagine receiving a medical diagnosis from an autonomous AI agent, as depicted below. Complex systems like the human body require nuanced and careful treatment, and AI overconfidence can be catastrophic in situations where human lives are at stake.

The path to reliable AI goes through principled probabilistic reasoning, often called Bayesian reasoning or uncertainty quantification. Uncertainty estimates can inform the AI agent when to defer to human expertise, e.g., in the case of medical diagnosis.

However, it comes at a price! Adding reliability to AI generally adds computational overhead associated with uncertainty estimation. Often this involves Monte-Carlo methods, which involve sampling from a high-dimensional probability distribution. The challenge of high-accuracy sampling has notoriously been highlighted for Bayesian neural networks, where researchers have essentially argued that high accuracy is both crucial for performance as well as computationally intractable with current digital hardware.

Our vision for Thermodynamic AI hardware aims to solve this key problem, enabling fast sampling of complex, high-dimensional probability distributions, in order to unlock scalable, reliable AI. 

We envision that the introduction of this innovative hardware paradigm will facilitate the development of novel machine learning applications. Through a feedback codesign loop, the collaboration between hardware and software will lead to their synchronized evolution, ultimately giving rise to advanced algorithms that would have been infeasible to execute on alternative hardware platforms. This concerted effort will enable us to explore uncharted territories in the realm of machine learning and push the boundaries of what can be achieved in this field.

The Stochastic Processing Unit (SPU)

With this motivation in mind, let us shift our attention to viable approaches to Thermodynamic Hardware. Our initial perspective on Thermodynamic Hardware, which we call the Stochastic Processing Unit (SPU), can be viewed as an electrical version of a coupled oscillator system. (See figure below for a mechanical analog.)

Each unit cell (as shown at the beginning of this blog) is an electrical oscillator, and they can be coupled together through resistive or capacitive coupling. The overall dynamics can be understood as a Langevin process and can be modeled with coupled stochastic differential equations, with free parameters that correspond to the particulars of the underlying electrical components. 

Digital discretizations of Langevin dynamics are commonly used in Markov Chain Monte Carlo (MCMC) sampling algorithms. We are essentially removing the discretization and running the algorithm in continuous time on the SPU hardware. By allowing the SPU hardware’s parameters to be freely programmed, the user gets to choose the probability distribution that they wish to sample from. 

The key difference with digital hardware is that, within the SPU, the samples are generated through the natural dynamics of the system, meaning they potentially could be generated faster, cheaper, and more energy efficiently.

With this in mind, let’s take a look at the playground!

Sampling with the Thermo Playground

Having introduced the SPU, let us now explore the results that it produces in the playground! 

Suppose that you wish to sample from a Gaussian distribution, which is a crucial task for reliable AI (not to mention other fields like finance and forecasting). In the case of a 2D Gaussian, our playground allows you to choose the covariance matrix of the distribution, as well as the SPU hardware parameters like sampling time, temperature, resistance, and inductance. 

The overall interface looks like this:

This interface consists of three key pieces, which are broken down below: (1) the knobs, (2) the 2D sampling plot, and (3) the 1D time-series traces. Let’s first start with the knobs: 

The user can adjust the sampling time, which is the time the hardware waits between gathering a new sample. Choosing the sampling time to be too short can lead to correlated samples, which is generally undesirable, since the samples should ideally be independent. The user can also adjust the temperature, which is directly related to the amplitude of the stochastic noise. As one can see in the animation, higher temperature leads to more stochastic jumping, which often leads to better sampling since it tends to lead to samples which are less correlated. The resistance and inductance knobs allow the user to adjust the parameters of the SPU circuit. Resistance tends to damp out stochastic fluctuations, so increasing resistance can lead to more deterministic trajectories, and hence more correlated samples. Inductance is analogous to mass or inertia, and tends to slow down the overall dynamics.

The 2D sampling plot gives a visual representation of the target distribution (the solid red circle represents the covariance matrix) and the generated samples (shown as black dots). In addition, the projections of the 2D samples onto the 1D axes are shown as histograms on the top and side of the plot. For comparison, the marginals (i.e., the 1D distributions) associated with the target distribution are shown as solid curves, overlaid with the histograms.

The 1D time-series traces show the two components of the samples in real time. A maximum of 1000 samples are allowed in the time window. Hence once this threshold is reached, old samples are discarded as new ones come in. In addition, there is a shaded box overlaying with the time-series trace, which illustrates the theoretical correlation time, i.e., the time over which samples are correlated. For ideal behavior (independent samples), one would want this shaded box to be small.

So that is the heart of the Thermo Playground! Now let us put this in the broader context of reliable AI.

Overconfidence of traditional AI approaches

Employing Large Language Models (LLMs) to high-stakes use cases such as medical/legal advice or enterprise workflows has yet to be fully unlocked due to their overconfidence and lack of reliability. 

To get some appreciation of what overconfidence looks like in AI, let us consider a simple toy example of classifying the well-known 'two moons' dataset, a dataset of two intertwined half-circular shapes, resembling two crescent moons. The task is to train a classifier to distinguish between the 'left moon' and the 'right moon.' In the figure below, these are represented as blue and orange dots.

ResNet, a popular convolutional neural network, can solve this task with high accuracy. When presented with in-distribution data points (those that lie on the crescents), it classifies them accurately, as seen in the figure below (left plot). 

However, challenges arise when we consider out-of-distribution (OOD) data points, those that do not lie on either of the two moons but in the empty space around them. The ResNet model will still make a confident prediction on OOD data, attributing them to one of the two moons with high certainty. This is shown in the figure above, where the test points in red are predicted with zero uncertainty to belong to the same class as the orange moon.

Let us now see how to resolve this issue of overconfidence on OOD data.

State-of-the-art Uncertainty Quantification

For classification with neural networks, a promising solution for dealing with the overconfidence is Spectral Normalized Gaussian Processes (SNGPs), which provide distance awareness (i.e., awareness of how far data points are from one another). SNGPs are an innovation on traditional Gaussian Process (GP) models that bring the benefits of uncertainty estimation without a significant increase in computational complexity. GPs express uncertainty naturally - they do not just predict a single outcome but rather a distribution of potential outcomes. However, GPs can be computationally expensive, limiting their practical applications. For more details on SNGPs, check out this tutorial.

A key subroutine in SNGP is the evaluation of the softmax Gaussian posterior, using either the mean field approximation or Monte Carlo estimation. In many applications the mean-field approximation is utilized. In this tutorial however, we use Gaussian sampling and Monte Carlo to evaluate the posterior, as our hardware is naturally suited to this task.

Below, we see that in our 'two moons' classification task, using SNGPs lead to a different outcome. For in-distribution data, the model makes confident predictions; however, for OOD data, rather than making an overconfident and likely incorrect prediction as a ResNet model might, the SNGP outputs a high uncertainty, as we can see for the region with the red points.

SNGPs in the Thermo Playground

Now check this out: the SNGP example (discussed above) is integrated into the Thermo Playground! This allows the user to play with the parameters of the SPU hardware and directly see the impact on the SNGP performance! The following animation shows how the user can play with the SNGP example inside of the Thermo Playground:

For example, the user can adjust the temperature of the stochastic noise. Increasing the temperature will reduce the correlations between samples and lead to a better overall performance of SNGP. Conversely one can see the performance starting to break down when lowering the temperature. We can also adjust the number of samples gathered, with more samples also leading to better SNGP performance.

Note that, here, we are using a simulator of the SPU to generate the samples required by the SNGP. Specifically, these are samples from a gaussian distribution whose covariance matrix is the output from the GP applied during SNGP. These samples are then used to estimate the uncertainty in predictions. So, we have shown how the SPU can be used in the context of machine learning to predict uncertainty in a simple classification task.

Below, we will discuss the exciting possibility of speedup for this type of application. 

Expected Performance Advantage with SPUs

We now focus on the possible speedup one could expect from using Thermodynamic Hardware relative to state-of-the-art GPUs. Here we consider the ideal case of precise electrical components and assume the unit cells in the SPU are fully connected.

To quantify the runtime and energy consumption of the Thermodynamic Hardware, we consider the effect of three key stages: digital compilation, loading/readout and the integration time of the physical dynamics needed to generate the samples. We assume the SPU is constructed from standard electrical components operating at room temperature. For this analysis, the user provides a precision matrix (the inverse of the covariance matrix), which is then compiled to the SPU in a digital pre-processing step, and then the dynamics of the SPU are run for the time needed to generate 10,000 samples.

To get a realistic picture of the potential advantage of the SPU it needs to be compared against some of the best available digital counterparts. Therefore, digital timing/energy results below were obtained using JAX run on an A100 GPU.
Below we show how the time taken to produce samples from a multivariate Gaussian scales with dimension. One can see that the SPU performance is expected to outperform the GPU at all dimensions. This speedup becomes exciting for high dimensional problems where we see an order of magnitude improvement.

The right hand side shows the breakdown in timings for the model of Thermodynamic Hardware when generating 20,000 dimensional samples. Interestingly, we expect the dominant step to be digital compilation. Improvements in this step would further increase the expected speedup.

Speedup is not the only thing to consider. Using the natural dynamics of a physical system massively reduces the expected energy requirements. Below we show how the energy of generating these samples is expected to scale with dimension relative to the JAX implementation. 

Here we see that the performance of the SPU really begins to shine: several orders-of-magnitude in energy cost improvement are expected for high dimensional problems.

Overall, a simple model of the SPU, the timings involved in its end to end operation, and the energy cost during these processes lead to a potential speedup of an order of magnitude and energy savings of several orders of magnitude or more for this task.

Algorithmic Co-Design

We emphasize that the SNGP approach and other uncertainty-quantification (UQ) approaches were developed and optimized for deployment on digital hardware, and it is not surprising that only modest speedups are expected for these digital algorithms. This suggests an exciting possibility of co-designing the AI algorithms together with the Thermodynamic Hardware. This co-design could unlock larger speedups and energy savings than those predicted for standard digital algorithms.

In what follows, we emphasize that looking beyond SNGP towards more complicated forms of uncertainty quantification, such as Bayesian neural networks, will likely unlock even larger speedups.

The Non-Gaussian Frontier

Moving beyond Gaussian sampling one can imagine using thermodynamic hardware to produce samples from more complicated distributions, such as the posteriors of Bayesian neural networks.

In Bayesian deep learning, the model's weights follow intricate probability distributions that need constant updating during training. This process requires generating multiple samples, which makes training Bayesian neural networks computationally demanding. Considering large models with millions of parameters, sampling from such high-dimensional and complex probability distributions becomes an arduous task. In practice, hundreds of Tensor Processing Units (TPUs) may be required for accurate sampling, e.g., with Hamiltonian Monte Carlo (HMC).

We can use similar timing models to those above to investigate when one would expect a ‘thermodynamic advantage’ for this application. Using our timing model and a JAX implementation of HMC leads to the following expected speedup from an SPU tailored to beyond-Gaussian sampling.

Our model suggests that for this application, one could achieve a speedup of several orders of magnitude, with the speedup generally increasing with dimension. The potential implications would be unlocking large-scale approximation-free Bayesian deep learning. This would be a key step towards the vision of the automated statistician and automated deep learning, transforming standard deep learning into a reliable, trustworthy application.

Outlook for Thermodynamic Advantage

Generative AI technologies like Large Language Models (LLMs) suffer from hallucinations and overconfidence. This prevents GenAI from being deployed in high-stakes use cases, such as enterprise workflows or applications where human lives are at stake, like medicine, transportation, or national security. 

Principled probabilistic reasoning (Bayesian methods), as we’ve discussed above, can solve these issues for GenAI. Bayesian methods output a probability distribution instead of a specific answer. Using that distribution we can compute the uncertainty of a text, say, in the context of text generation.  It turns out that high uncertainty implies a likely hallucination. This gives us a direct way to detect hallucinations for LLMs or other GenAI technologies.

In this blog, we have illustrated uncertainty quantification with simple methods on a toy dataset. But ultimately, a key technological breakthrough would be large-scale reliable AI, including reliable LLMs. Achieving this goal will ultimately involve increasing the complexity of uncertainty awareness, involving methods that go beyond mean-field approximations. This is where Thermodynamic Hardware is so crucial. Simplistic, approximate methods for uncertainty quantification can be employed on today’s digital computers. But complex, high-accuracy methods are prohibitively slow, and Thermodynamic Hardware could unlock these methods.

In the figure below, we illustrate the idea of a hierarchy of uncertainty awareness. This hierarchy includes, in order of increasing complexity: mean-field variational inference (MFVI), SNGP, stochastic gradient Markov chain monte carlo (SGMCMC), and HMC. As we increase the complexity of uncertainty awareness, the potential performance gain (runtime speedup and energy savings) from Thermodynamic Hardware increases.

We emphasize that this is just a schematic plot, and there remain significant engineering challenges to be overcome to realize the large-scale hardware required for complex uncertainty awareness. 

Nevertheless, the future is bright for Thermodynamic Hardware and reliable AI.

Step into the Future with Normal Computing

We invite you to try out the Thermo Playground! We hope this gives you a feel for programming actual Thermodynamic Hardware. Finally, we hope that our vision of reliable AI resonates with you.  

We thank you for taking the time to read this blog post! Be sure to sign up for updates at Normal Computing’s website.

Our preprint Thermodynamic Linear Algebra, submitted on August 10th, drew a warm response from the broader tech community, stimulating thoughtful questions on Hacker News. Yann LeCun also apparently found it “Fun”, and some people started betting on a hardware implementation of our proposal.

This blog post will cover several highlights from the paper, including:

• Novel thermodynamic algorithms for four different linear algebra primitives.

• The first asymptotic scaling analysis for the thermodynamic computing paradigm, and the first theoretically predicted speedup for a thermodynamic algorithm.

• The predicted speedups grow with the difficulty of the problem, i.e., with the matrix dimension and matrix condition number, and hence the speedups can be increased by considering more difficult problems.

• Numerics with a detailed timing model for thermodynamic hardware, allowing comparison of predicted runtimes in practice, and indeed these numerics show that speedup is possible with realistic hardware in realistic situations.

• A fundamental relation that captures the tradeoff between energy and time in algorithmic complexity of thermodynamic algorithms. This tradeoff may be relevant more broadly (e.g., for digital methods) and impacts the cost and sustainability of AI services.

• The potential for using thermodynamics as a lens on computational complexity, which is a new perspective on the optimal performance of algorithms.

In this blog post, we dive deep into these results, and provide context for their interpretation. This includes a historical context, since after all, thermodynamics and linear algebra are two iconic fields, each with their own complex histories. In what follows we first discuss this epic history.

Linear algebra, from application to hardware

Ancient History

The ancient text Jiushang Suanshu (The Nine Chapters on the Mathematical Art) was compiled in the span of 800 years beginning in the 10th century BC, and contains the following problem (the details of which are non-essential, but we include for the interested reader):

Suppose we have 3 bundles of high-quality cereals, 2 bundles of medium-quality cereals and one box of poor-quality cereals, amounting to 39 dou of grain; [suppose we also have] 2 bundles of high-quality cereals, 3 of medium-quality and one of poor quality, amounting to 34 dou of grain; one bundle of high-quality cereals, 2 of medium quality and 3 of poor-quality, amounting to 26 dou of grain. Question: how many dou of grain in 1 bundle of high-, medium- and poor-quality cereals, respectively?

Answer: 1 bundle of high-quality cereals: 9 dou 1/4; 1 bundle of medium-quality cereals: 4 dou 1/4; 1 bundle of poor-quality cereals: 2 dou 3/4.

This problem is, in modern terminology, a linear system of three equations, which represent constraints on the amount of grain contained in a single package, for various grades of cereal. Its solution may be depicted as the intersection of three planes, as shown in the animation below.

In Jiushang, methods involving a set of bamboo “counting rods” are presented, which were used to implement a set of algorithms that formed the rod calculus. Using these methods, a clever farmer might ascertain the content of a box of grain without measuring, and maybe save time. The text therefore exemplifies some of the earliest applications, hardware, and algorithms for solving linear systems of equations.

From cubic to quadratic scaling

Nowadays agriculture is just one of many places where linear systems appear. The applications for linear algebra problems have multiplied over the years, and so have the means of solving them. Homo Sapiens, as a species, has a reputation for its innovative use of tools, and, in a way, the tools used in different times even define the eras of civilization. Bamboo rods worked well for linear systems with only a few unknown quantities, but quickly became impractical as the number of variables grew.

The standard method used to solve linear systems with counting rods seems to have been equivalent to Gaussian elimination, meaning the number of steps in the calculation was proportional to the cube of the number of variables. Currently, the state-of-the-art tool is a digital computer with a Graphics Processing Unit (GPU), a chip which is specifically designed to quickly perform linear algebraic operations. Instead of using Gaussian elimination, other algorithms are available (such as conjugate gradients) to be run on the GPUs which only require a number of steps proportional to the square of the number of variables, a great improvement over the ancient methods.

Linear algebra in machine learning

One area where GPUs have massively contributed to accelerating linear algebra primitives is machine learning. One of the many examples of solving a linear system of equations in machine learning is in support vector machines. There, one must find the hyperplane that best separates two classes of features. Below is a figure where the plane corresponding to the solution of the linear system is shown, that best separates blue and green sets of points which are often extracted features of a dataset.

Another example is Gaussian process regression (GPR), which is an important algorithm that includes uncertainty estimates. In GPR, one must invert the kernel matrix to obtain the mean and covariance of the posterior probability density function. There, matrix inversion is one of the more costly subroutines of the algorithms. A third linear algebra primitive we focus on in our paper is estimating the determinant, which is also an important subroutine in Bayesian learning.

Evolution of hardware

What our paper is about is a new physics-based tool for solving linear-algebraic problems, along with new algorithms to use with it. This physics-based system can be viewed as a thermodynamic computer.

The thermodynamic computer

Thermodynamics is often called a theory of work and heat, which it is. But as conceived by its “father” Sadi Carnot, it was a theory about steam: in particular, of how to harness its power, to capture steam in a piston and use it to drive hulking locomotives across Europe. The subject does not find its origins in intellectual curiosity, and definitely not in a philosophical investigation of time’s arrow, but in the need to solve a practical problem. For this reason, thermodynamics has a different flavor than other physical theories, because it is more about how a system responds to a procedure conducted by an external operator, rather than the behavior of the system in isolation. As Herbert Callen writes in Thermodynamics and an introduction to thermostatistics:

The single, all-encompassing problem of thermodynamics is the determination of the equilibrium state that eventually results after the removal of internal constraints in a closed, composite system.

In fact, much of the theory can be interpreted as relating the inputs and outputs of a system undergoing an engine cycle, which is a sequence of operations that are performed on the system, one after another. Consider, though, that a sequence of operations performed on a device, resulting in a particular relation between its inputs and outputs, is nothing other than an algorithm. In other words, thermodynamics is the study of physically realized algorithms operating on macroscopic systems.

How, then, do we design a thermodynamic device to implement algorithms that are useful? One way is to use the minimum energy principle: that a system in thermal equilibrium with its surroundings is in a macroscopic state where the energy is minimized with respect to all external constraints on the system (including temperature). If the microscopic state of the device is described by a vector of coordinates x and the potential energy of the system is given by a function V(x), then at thermal equilibrium the coordinate vector x is a random variable, whose density function is the Boltzmann distribution

where Z (the partition function) is a normalization factor. Then we need only engineer the potential energy V(x) such that this equilibrium distribution somehow contains the solution to the problem we would like to solve. Suppose that we take a potential energy function of the following form

where the (positive definite) matrix A and vector b are chosen by the user. In this case, the Boltzmann distribution is a normal distribution, or Gaussian

The mean of this distribution is the solution to the linear system Ax = b. We also see that the covariance matrix is proportional to the inverse of the matrix A. Using these two facts, we can devise a protocol for solving linear systems and inverting matrices based on the thermodynamics of this system, which physically corresponds to a set of harmonic oscillators that are coupled to each other.

The method is illustrated in the figure below. One simply sets the matrix A and vector b to those appearing in the equation Ax = b, then, after waiting for equilibration, one can average the value of the vector x over time using, for example using an electrical circuit, where the components of x might be the voltages at different points in a circuit. This average will be the solution of a linear system, and similarly the covariance matrix of x can be estimated by an analog device.

While it may be interesting in an academic way that this is possible, it is not immediately clear that there is any reason to attempt to solve these problems using networks of coupled oscillators, especially given that the SOTA GPUs are so effective. Remarkably, the simple method described above can be theoretically shown to be more efficient than existing digital methods in the limit of a large number of variables.

Achieving thermodynamic advantage

Thermodynamic speedups

Efficient ways to solve linear algebra problems were largely discovered in the 20th century, motivated by the need to solve larger problems with the limited hardware available at the time. These include, for example, the conjugate gradient method for solving linear systems and the Cholesky decomposition, which can be applied to linear systems, matrix inversion, and other problems. Even with these methods, the demand grew for faster and faster calculations; the live rendering of the three-dimensional environments needed for modern video games was especially taxing, spurring the invention of the GPU. Modern GPUs accelerate the evaluation of linear functions significantly, but the number of algorithmic steps needed to solve a linear system of equations has remained asymptotically proportional to the square of the number of variables, while inverting a matrix has required a cubically-growing number of steps.

This is often expressed using big O notation. When we write that a runtime is in O(d), it means that the runtime scales linearly with dimensionality, omitting any prefactors in what the actual runtime is, that can depend on other factors such as parallelization.

The thermodynamic algorithm for solving linear systems outlined previously takes advantage of the physical equilibration process in the hardware, which can be thought of as performing matrix-vector multiplications in constant time. This enables us to prove that a linear system of equations can be solved in an amount of time proportional to the number of variables using a thermodynamic computer. We derived similar algorithms for solving other common problems in linear algebra: inverting a matrix, solving the Lyapunov equation, and evaluating the determinant of the matrix. In each case, the asymptotic time-complexity of the thermodynamic algorithm is better than the best known digital algorithm by a factor on the order of the number of variables, which can probably be explained as stemming from a constant-time matrix-vector multiplication achieved by the hardware.

Time and energy

It could be argued that the number of steps taken by a digital algorithm is not the right metric, given that these algorithms may be parallelized sometimes, so the physical time necessary is not limited by the number of elementary operations. Another cost of computation is the necessary energy expenditure though, which should also be taken into account. If parallelizing an algorithm across two processors decreases the time, it may also increase the energy, so the product of energy and time is of special importance. The thermodynamic algorithm for solving a linear system has energy-time product scaling with the number of variables, whereas all existing digital algorithms we are aware of have energy-time product scaling with at least the square of the number of variables. Our paper gives the energy-time complexity for solving linear systems is as

where d is the number of variables, κ is the condition number of A (its largest eigenvalue divided by its smallest eigenvalue), and the other quantities are physical constants and tolerance parameters specifying the degree of accuracy of the solution.

Harder, better, faster, stronger

We’ve reached a milestone in this story–the thermodynamic computer is a new piece of hardware that can solve linear algebra problems faster than conventional digital computers, under mild assumptions. Will there be further progress, or is this paradigm the optimal model of linear algebraic computation? It is easy to imagine a future in which incremental improvements in efficiency continue, relegating the thermodynamic computer to the hardware graveyard alongside bamboo counting rods. But underlying this question is a deeper one about the fundamental hardness, or complexity, of solving the problem at hand. It is relatively straightforward to prove that a particular problem can be solved with an amount of resources limited by some upper bound; one must simply provide the algorithm. But lower bounds on computational hardness are far more elusive, and finding such bounds remains a sticking point of computational complexity theory.

In thermodynamics, proven lower bounds on the resources necessary to accomplish some task are commonplace, for example the minimum energy which must be dissipated in order to produce a certain amount of work from two heat reservoirs is the seminal result of Carnot’s study of engine efficiency. In fact, thermodynamics is the only branch of physics where we have a definite notion of irrecoverable cost, and the general format of these results is that to do a given thing in a shorter period of time, one has to spend more energy. So, perhaps, the thermodynamic computing paradigm can shed light on the complexity of computations. This idea goes hand in hand with the study of optimality for thermodynamic algorithms. In calculating the efficiency of his engine cycle, Carnot found that this result was not just relevant to steam engines, but in fact was a general limit on the efficiency of any device which produces work from heat. Similarly, we conjecture that the limit we have found on energy and time costs may be generalizable to other models of computation. Because our results can be stated in terms of the energy-time product necessary for a computation, it is natural to investigate whether thermodynamic algorithms exist that achieve a lower energy-time product (in which case our algorithms would not be optimal), or otherwise to prove that none exist.

This line of thinking connects to the “mortal computation” advocated by Geoff Hinton (which inspired the title of this post). While cutting-edge artificial intelligence algorithms consume great quantities of energy, it is not known whether they may be carried out at reduced cost, and to what extent, by analog hardware that offloads some or all of the algorithm to naturally-occurring physical processes. It seems important, then, to ask whether thermodynamic formulations can identify the optimal efficiencies of AI algorithms, and ultimately provide protocols for achieving optimal performance.

Speedups: in practice

You may also be curious of what speedups we can actually get in practice, other than bounds, that may not be tight and which put aside prefactors. To investigate this, we compared our method’s performance by measuring the error obtained by running it on ideal hardware (with no imperfections, that we leave for future work) to the performance obtained for other digital methods. To solve linear systems, the conjugate gradient method and the Cholesky decomposition are used. We also estimated the runtime of the thermodynamic hardware by constructing a timing model that takes into account all the operations involved to obtain a solution, such as digital-to-analog and analog-to-digital conversions, and other digital overhead. This model is based on an RC circuit implementation that only involves passive electrical components, and is discussed in the next section. The results are that even for modest dimensions (around 1000) and condition numbers, we obtain solutions competitive to conjugate gradients. We see that this advantage grows with dimension, as expected from the analytic bounds we obtain. We also see that by lowering the temperature, even better results can be obtained, hence showing the thermodynamic nature of the device.

This is even more exciting, as it shows that a simple thermodynamic hardware proposal can yield speedup advantages for simple problems for which GPUs were optimized for. Hence we expect this to be optimized a lot in the future.

A potential hardware realization

The discussion up to this point may seem very abstract–what does a thermodynamic computer actually look like “in the flesh”? Below is a circuit diagram of one possible implementation of a single component (or cell) of the thermodynamic computer, taken from our previous work (see also this talk).

To solve a linear system that has d variables, d of these cells would be needed, with each cell coupled to all the others (for d² total connections). A single cell contains a resistor and a capacitor, and a noise source in series (the resistor also produces voltage fluctuations due to Johnson-Nyquist noise at finite temperature, which may be effectively included in the noise source). There is also a constant DC bias applied to each cell, which encodes the components of the vector b. The entries of the matrix A are encoded in the couplings between cells, which may be either resistive or capacitive. Of course this is just one possible version of the thermodynamic computer; recall that the definition of the device is at the level of the potential energy function V(x), and the vector x may represent essentially any measurable physical degree of freedom. As all physical systems have energy functions (Hamiltonians), this is a very general level of description, which could be applied, for example, to engineering similar models with optical hardware.

We thank you for taking the time to read this blog!

A bright future

At the core of Normal Computing’s mission is bridging artificial intelligence, particularly generative AI, to high stakes enterprise decision-making applications. We are approaching these problems with a mix of interdisciplinary approaches across the full stack, from hardware and physics to software infrastructure and algorithms. Future physics-based hardware may allow for a step change in AI reliability and scale, in part by admitting new co-developed algorithms. If you are interested in pushing the boundaries of reliability in AI with us, get in touch at info@normalcomputing.ai!

Unreasonably effective?

Eugene Wigner famously stated that mathematics is unreasonably effective in the natural sciences. He noted that, “the enormous usefulness of mathematics in the natural sciences is something bordering on the mysterious and that there is no rational explanation for it”.

In machine learning and AI, it could similarly be argued that physics-based models are unreasonably effective. Foundational principles from thermodynamics and classical physics are routinely leveraged in modern AI and statistical learning algorithms such as diffusion generative models [1], Monte Carlo sampling methods based on Hamiltonian dynamics [2], and optimization algorithms inspired by physical annealing [3]. When referring to AI here, we note that we broadly encompass classical statistical algorithms in addition to, e.g., contemporary deep learning-based AI methods.

In this blog post, we will take a closer look into why there seems to be a mysterious synergy between physics and AI. Why are some of the more successful AI algorithms inspired by physics? Is this just a coincidence, or does it reveal a more profound, fundamental connection between the two?

Our main goals

It is clear that many modern AI models are not explicitly motivated by physics, such as transformers for language modeling (however, see here for a compelling connection). Hence, physics is clearly not necessary for performant AI. On the other hand, in many contexts, physics-based models are competitive with non physics-based ones, suggesting that physics may often be sufficient as a guiding principle. For example, state-space models, which were originally employed for control engineering, draw inspiration from physics and were recently shown to be competitive for language modeling [30]. Such sufficiency would nevertheless be compelling, since it motivates the development of highly physical approaches to AI algorithms with implications, e.g., for designing physics-based AI hardware. Thus, one goal of this blog is to highlight several examples where physics-based reasoning led to the development of new AI algorithms or new perspectives on existing algorithms.

The second goal is to provide explanations for why physical reasoning is often empirically helpful in providing high-level guiding principles for AI algorithm development. We will highlight features of physics-based models, such as continuity, interpretability, symmetry, and simplicity, which make them useful in certain real-world applications.

Some features of physics-based models

Simplicity as a feature

Occam’s razor: Physics-inspired models are often inherently simple and parsimonious, since physics is concerned with revealing simple universal truths; this has led to the description of physics as a “search for simplicity”. The simplicity of physics-based models is often considered a feature rather than a bug, which one can understand from the perspective of Occam’s razor: the guiding principle that advocates for seeking explanations constructed using the fewest possible components. The effectiveness of Occam's razor has been demonstrated time and again across various disciplines, as simpler explanations often prove to be more robust, reliable, and generalizable.

One can similarly apply Occam’s razor to machine learning models, where the goal is often to seek high-level explanations for seemingly complicated datasets that nevertheless generalize well. Physics-inspired modeling can therefore be viewed as an attempt to implicitly leverage Occam’s razor in the context of machine learning.

Caption: Natural data (left) is often embedded in a lower-dimensional manifold (right) in a structured but non-obvious way. Physics-grounded methods can be effective in unearthing this structure.

Low-dimensional manifolds: The manifold hypothesis [4] can be seen as a manifestation of Occam’s razor. The statement is that most natural datasets lie close to a low-dimensional manifold, which happens to be embedded in a higher dimensional space in a structured way. (See figure above where the data lie on a torus manifold embedded in 3D space.) The hypothesis alludes to the existence of simpler explanations of the data, waiting to be found.

In this context, the manifold hypothesis provides further motivation for physics-based models: physical laws give rise to low-dimensional manifolds of natural data, and physics-inspired models serve as powerful tools to effectively unearth explanations behind real-world datasets.

Smoothness as a feature

Mathematics is the Wild Wild West, where anything goes. Discontinuous and non-differentiable functions abound. In principle, one could search over the space of such pathological (colloquially speaking) functions during the learning process in the context of AI. 

Oftentimes, explicitly regularizing the learning process to prefer smooth, continuous function is desirable. Physics-based methods tend to be biased towards well-behaved functions since they have a correspondence to the natural world. Thus, choosing a model inspired by physics can be seen as applying an inductive bias to the statistical learning process, which can be useful for speeding up training and inference as well as improving the generalizability of models. Indeed, the smooth and well-behaved properties of physics-based models have recently been exploited to construct novel classes of physics-based generative AI models [5].

Symmetry and scale separation as features

As mentioned above, natural datasets often live in structured, low-dimensional manifolds. The use of physically-informed inductive biases can place powerful priors on the space of functions that AI models process, helping effectively break the curse of dimensionality associated with complex real-world data. Two such physics-informed inductive biases are symmetry and scale separation. Scale separation enforces a multi-scale, hierarchical structure ubiquitous in the real world, common examples being the pooling operation in convolutional neural networks and the philosophy behind wavelet-based approaches to data analysis.

Phil Anderson famously said that “it is only slightly overstating the case to say that physics is the study of symmetry”. Symmetry priors are used as guiding principles behind many neural network models today [11]. Convolutional neural networks, for example, naturally emerge through enforcing translational symmetry [12]. Another prominent example is AlphaFold 2, where the use of 3-D symmetry was instrumental to achieving its paradigm-shifting performance for protein structure prediction [13]. 

Physics-based inductive biases are especially useful when analyzing data from physical systems, and this has motivated neural networks based on Hamiltonian [14] and Lagrangian [15] dynamics. Taking this one step further, one can literally use a physical system as the neural network, as in deep physical neural networks [16]. All of these ideas can be exploited from an inductive bias perspective.

Caption: Symmetry-preserving models (right) can faithfully incorporate physical transformations of the data (left) unlike vanilla models (middle).

Interpretability as a feature

Interpretability is often a desirable feature of AI models. Since humans have strong intuition for physical systems, the parameters of a physics-based model can carry conceptual meaning. Closely related to the concept of symmetry priors above, restricting the nature of message passing in graph neural networks in a domain-specific manner can result in learning interpretable features, as in this example of learning the dynamics of gravitationally-bound multi-particle systems using symbolic expressions for message passing. Interpretability thus  provides another motivator for physics-based models, further highlighted in [6,7].

Besides feature interpretability, physics-based algorithms can also aid in model interpretability. Algorithms used to fit AI models frequently have complex behaviors and physics can be a key tool in understanding how they work. For instance, the physics-based intuition that adding momentum to a model will aid in exploration is born out in both the popular Nesterov-type extensions to stochastic gradient descent [31], and in the success of Hamiltonian Monte Carlo [2]. In the latter case, the simple physics-based intuition that the Hamiltonian should stay constant when generating a proposal leads to both an extremely effective algorithm for sampling from a general probability distribution, but also a natural model diagnostic (has the value of the Hamiltonian diverged from its initial value?) that can successfully indicate that your model has problematic features.

Dynamical modeling as a feature

Explicitly modeling the learning process as a continuous dynamical system opens up a range of new possibilities, such as leveraging powerful differential equation solvers and being able to effectively model continuous underlying latent processes. A canonical example is that of Neural Ordinary Differential Equations (Neural ODEs) [8], which introduced a class of AI models where a pass through the network involves a continuous transformation -- the network is explicitly modeled as a dynamical system (Figure below from [8]).

Caption: Neural ODEs can explicitly model neural networks as dynamical systems (from [8])

A key motivator was computational: differential equation solvers, which had been developed and fine-tuned over many decades, could be leveraged for much of the computational heavy lifting. Neural ODEs and their variants show promising performance on a variety of tasks across domains and enable principled inference for datasets that are challenging to analyze with traditional methods, such as irregularly-sampled time series.

Since physical processes are often governed by differential equations, neural ODEs encourage an analogy between neural networks and continuous-time, physical dynamical processes. This can allow for the discretization of the underlying differential equations in a way that can be used to efficiently realize neural networks at the hardware level.

A recent, related line of work explicitly re-interprets graph neural networks and associated learning processes as continuous dynamical systems inspired by physics. This provides deep insights into the nature of learning on graphs that are otherwise difficult to address with traditional graph frameworks by, e.g., drawing direct analogies between different types of GNNs and diffusion PDEs solved by specific numerical schemes.

Probabilistic nature a feature

Probabilistic machine learning [9] is becoming increasingly prevalent, given the need for principled uncertainty quantification across domains as well as the real-world impact of generative modeling over complex data distributions, from natural images to biomolecules. A probabilistic approach to AI allows for principled uncertainty quantification and increases the reliability and trustworthiness of the model’s predictions while leveraging powerful concepts from traditional statistical inference, such as latent variable modeling.

Caption: Probabilistic methods (here: Gaussian processes) can effectively model distributions over quantities of interest, including uncertainties.

Not surprisingly, the field of probabilistic AI in particular has benefitted from many physics-inspired models (some of which we discuss below). This is partly because probabilistic reasoning is prevalent in physics, especially in statistical mechanics, thermodynamics, and quantum mechanics. These subfields of physics therefore offer both intuition and formal frameworks for formulating models for probabilistic AI.

Examples of physics-inspired models

Having outlined several features of physics-based approaches, let us highlight some key examples where these approaches have yielded powerful AI and statistical learning algorithms in practice.

Energy-based models

It may be evident, but energy-based models (EBMs) [19] draw substantial inspiration from physics. Specifically, Hopfield networks and Boltzmann machines were originally inspired by spin-glass models, which are Ising spin models with random couplings. EBMs like Boltzmann machines can be used for sampling  in generative AI applications, highlighting a more general trend of physics-based being especially useful for generative AI.

Diffusion models

Perhaps more than any other model, diffusion models [1] embody the spirit of this message: they are probabilistic, they are continuous in time [21], and of course, they are physics-inspired. Originally inspired by non-equilibrium thermodynamics [1], diffusion models incorporate noise into the learning process, effectively smoothing the distribution to facilitate learning. This is particularly useful given the manifold hypothesis (see above), which suggests that learning natural data distributions would be challenging due to their complex structure.

Caption: Diffusion models are a class of physics-inspired generative models that can learn to iteratively generate complex data starting from noise.

Samples are then drawn by reversing the diffusion process, providing a state-of-the-art algorithm for generative AI that is used for applications including molecular docking [22], protein folding [23], and crystal design [24]. Recently, the connection to physics has been pushed even further [5], extending the generative AI paradigm to other physics-inspired models such as the Poisson equation in electrostatics.

Markov Chain Monte Carlo

In probabilistic AI and Bayesian inference more broadly, Markov chain Monte Carlo (MCMC) [25] is a workhorse for various learning tasks. The history of sampling algorithms like MCMC is closely intertwined with physics. The modern version of Markov Chain Monte Carlo was invented at Los Alamos in the 1940s by Stanisław Ulam while studying neutron diffusion in nuclear weapon cores, and further fleshed out in work with John von Neumann and Nicholas Metropolis. 

An important criterion in MCMC is detailed balance, connected to the invertibility of the sampling proposal distribution. Physics yields simple and elegant ways to enforce detailed balance, e.g. through energy conservation. Inspiration from physics has continued to yield rich classes of techniques for both optimization and sampling in recent years.

Hamiltonian Monte Carlo: A state-of-the-art MCMC algorithm is Hamiltonian Monte Carlo (HMC), which is based on introducing an auxiliary variable (“momentum”) and evolving particles in time according to Hamiltonian dynamics [2]. These dynamics, which stem from classical mechanics and are intimately related to the principles governing the motion of physical systems, facilitate inference by proposing samples that effectively explore the parameter space in a much more efficient manner than traditional MCMC algorithms like Metropolis-Hastings-Rosenbluth (Figure below).

Caption: Sampling algorithms based on Hamiltonian dynamics (top) can sample complex distributions much more efficiently than traditional MCMC methods (bottom).

Bolstering the theme of physics roots for sampling techniques, HMC (also known as Hybrid Monte Carlo) was originally proposed in the late 1980s in the context of studying lattice chromodynamics, a theoretical framework for describing the strong nuclear force that binds quarks and gluons together into protons, neutrons, and other particles.

Stochastic gradient HMC: Diving deeper, there is an esoteric but useful MCMC algorithm called stochastic gradient Hamiltonian Monte Carlo (SGHMC) [26]. Here, noisy estimates of the gradient are employed, which introduces stochastic fluctuations into the dynamics. These fluctuations are undesirable: they perturb the dynamics, and to deal with this dissipation was introduced into the equations to damp out the noise. In this case, fluctuations alone were detrimental, but combining fluctuations with dissipation leads to good properties in the solution. 

Intriguingly there is a theorem in thermodynamics called the fluctuation-dissipation theorem, which is typical of say Brownian motion, stating that fluctuations and dissipation always go hand-in-hand, like two peas in a pod. So it could be argued that the SGHMC algorithm is physics inspired, and in particular thermodynamics inspired.

Variational inference algorithms

Variational inference is a complementary method to MCMC for approximating complex posterior distributions, turning posterior inference into an optimization rather than sampling problem. Instead of trying to compute a complex distribution directly, the method defines a simpler, typically parameterized family of distributions, and then finds the member of that family which is closest to the target distribution.

There is a direct analogy with variational methods from quantum mechanics, where the idea is to find variational approximations to ground state wavefunctions of a system by minimizing the expectation value of the energy. On a historic note, exploiting the connection Ising models and Boltzmann machines initially introduced variational methods to the field of AI in the late 1980s, which are a cornerstone of today's probabilistic AI toolkit; see here for a deeper exploration of this connection.

Caveats

Now that we have presented some arguments and examples, in the spirit of steelmanning let us offer some caveats, to the skeptics out there, regarding the connection between physics and AI.

Historical and psychological effects

Since the field of physics was developed before AI, it could be argued that historical effects play a role. Indeed, physics has a long history even before the term AI was coined in 1955. As a result, researchers may be using their past experience in physics as their starting point for how to formulate models in AI. Moreover, human psychology plays a role: we have great intuition for physical processes since we interact with the physical world. Hence, our own intuition lends itself to physics-based modeling..

Is math the intermediary?

The fundamental underpinnings of both physics and AI are naturally formulated in the language of mathematics, including concepts from geometry, linear algebra, and information theory. Perhaps it is their mutual connection to these subfields of mathematics that effectively connects physics and AI?

When it comes to statistical learning, we are usually still grounded in the natural world, relying on concepts like geometric data types (scalars, vectors, tensors, …) and probability densities—even if we work with abstract models that obfuscate this connection. It may therefore not be entirely surprising if concepts that are ubiquitous in formulating physical principles, like geometry and linear algebra, work well in the realm of inference.

Similarly, the concept of entropy shows up in a central way in both thermodynamics and machine learning. But it could be argued that this is because both thermodynamics and machine learning are mutually connected to a third field: information theory, a mathematical framework for quantifying and propagating information. One can see how machine learning is connected to information theory since, e.g., data compression, latent variable modeling, and the information bottleneck are central concepts here. Hence, the unifying language of information theory happens to be a useful lens for both physics and machine learning.

A good starting point?

Physics-based models are clearly effective for building inference strategies. However, they may be most effective as starting points, with engineering and practical considerations taking over at some point. For example, Hamiltonian Monte Carlo is notoriously hard to tune in many cases, with domain- and problem-specific engineering and insight often needed to get it working well.

The above image was generated by the text-to-image diffusion model Midjourney, and hence diffusion models are clearly useful! The diffusion process in these models gives one way to destroy and create information, but there are many others, some physics-based (e.g. based on electrostatics), some not really (e.g. cold diffusion). The jury is still out on which types of models will strike a good balance in the end between all the relevant axes. 

There is clearly something very fundamental in why physics-based models work well for statistical inference and for AI. But they may be best as starting points and as ways to guide strategies for building models -- otherwise one could get stuck in a local minimum. However new physics-based strategies often lead to qualitative leaps in performance and understanding, providing fundamental breakthroughs at the conceptual level.

Hardware Implications

While algorithmic developers have been quick to pick up on the connection between physics and AI, the implications for designing AI hardware have not been fully appreciated. All computing hardware naturally obeys the laws of physics, including digital hardware. But in theory one could exploit the laws of physics more directly with analog hardware, where physical evolution is exploited as part of the computation. We therefore leave the intriguing question open: how can one exploit the aforementioned connections between physics and AI to better design AI hardware? Some recent work along these lines includes probabilistic hardware with p-bits [27,28] and thermodynamic hardware with stochastic units coupled to an entropy regulator [29], both of which have applications for probabilistic AI and generative AI.

Conclusion

In summary, we highlighted several positive features of physics-inspired models that make them a good fit for use in AI.

• Their parsimonious nature allows for an elegant trade off between expressivity and simplicity;

• Incorporating domain-specific parameterization can allow for greater interpretability of physics-based models;

• They allow one to naturally incorporate inductive biases such as symmetries and scale separation;

• Continuous-time formulations enabled by physics-based models inherit computational and other benefits; and

• They provide a compact, intuitive framework for probabilistic reasoning.

We provided numerous examples of the marriage of physics and AI, from energy-based models to diffusion models to Hamiltonian Monte Carlo to annealing-based optimization. We also pointed out some caveats like historical efforts that may exaggerate the impact of physics in AI. 

We thank you for taking the time to read this blog post! We hope this post stimulates discussion around the intersection of physics and AI. Be sure to sign up for updates at Normal Computing’s website.

Acknowledgments

We thank Anna Golubeva for helpful conversations and feedback.

References

[1] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015, June). Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (pp. 2256-2265). PMLR.

[2] Neal, R. M. (2011). MCMC using Hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11), 2.

[3] Kirkpatrick, S., Gelatt Jr, C. D., Vecchi, M. P. (1983). Optimization by Simulated Annealing. Science. 220 (4598): 671–680.

[4] Fefferman, C., Mitter, S., Narayanan, H. (2016). Testing the manifold hypothesis. Journal of the American Mathematical Society. 29 (4): 983–1049.

[5] Liu, Z., Luo, D., Xu, Y., Jaakkola, T., Tegmark, M. (2023). GenPhys: From Physical Processes to Generative Models. arXiv preprint arXiv:2304.02637.

[6] Takeishi, N., Kalousis, A. (2021). Physics-integrated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems. 34: 14809–14821.

[7] Zehtabiyan-Rezaie, N., Iosifidis, A., Abkar, M. (2023). Physics-guided machine learning for wind-farm power prediction: Toward interpretability and generalizability. PRX Energy. APS. 2 (1): 013009.

[8] Chen, R. T. Q., Rubanova, Y., Bettencourt, J., Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems. 31.

[9] Murphy, K. P. (2022). Probabilistic machine learning: an introduction. MIT press.

[10] Steinruecken, C., Smith, E., Janz, D., Lloyd, J., Ghahramani, Z. (2019). The automatic statistician. Automated machine learning: Methods, systems, challenges. Springer International Publishing. 161–173.

[11] Bronstein, M. M., Bruna, J., Cohen, T., Velickovic, P. (2021). Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478.

[12] O'Shea, K., Nash, R. (2015). An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458.

[13] Jumper, J. et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature. Nature Publishing Group UK London. 596 (7873): 583–589.

[14] Greydanus, S., Dzamba, M., Yosinski, J. (2019). Hamiltonian neural networks. Advances in neural information processing systems. 32.

[15] Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., Ho, S. (2020). Lagrangian neural networks. arXiv preprint arXiv:2003.04630.

[16] Wright, L. G., Onodera, T., Stein, M. M., Wang, T., Schachter, D. T., Hu, Z., McMahon, P. L. (2022). Deep physical neural networks trained with backpropagation. Nature. Nature Publishing Group UK London. 601 (7894): 549--555.

[17] Morris, E. R., Searle, M. S. (2012). Overview of protein folding mechanisms: experimental and theoretical approaches to probing energy landscapes. Current protocols in protein science. Wiley Online Library. 68 (1): 28–2.

[18] Ortega, P. A., Braun, D. A. (2013). Thermodynamics as a theory of decision-making with information-processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences. The Royal Society Publishing. 469 (2153): 20120683.

[19] LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data. 1 (0).

[20] Marullo, C., Agliari, E. (2020). Boltzmann machines as generalized Hopfield networks: a review of recent results and outlooks. Entropy. MDPI. 23 (1): 34.

[21] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.

[22] Corso, G., Stärk, H., Jing, B., Barzilay, R., & Jaakkola, T. (2022). Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776. 

[23] Wu, K. E., Yang, K. K., Berg, R. V. D., Zou, J. Y., Lu, A. X., & Amini, A. P. (2022). Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.15611.

[24] Xie, T., Fu, X., Ganea, O. E., Barzilay, R., & Jaakkola, T. (2021). Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197.

[25] Andrieu, C., De Freitas, N., Doucet, A., Jordan, M. I. (2003). An introduction to MCMC for machine learning. Machine learning. Springer. 50: 5–43.

[26] Chen, T., Fox, E., Guestrin, C. (2014). Stochastic gradient Hamiltonian Monte Carlo. International conference on machine learning. PMLR. 1683–1691.

[27] Camsari, K. Y., Sutton, B. M., & Datta, S. (2019). P-bits for probabilistic spin logic. Applied Physics Reviews, 6(1), 011305.

[28] Camsari, K. Y., Faria, R., Sutton, B. M., & Datta, S. (2017). Stochastic p-bits for Invertible Logic. Physical Review X, 7, 031014.

[29] Coles, P. J. (2023). Thermodynamic AI and the fluctuation frontier. arXiv preprint arXiv:2302.06584.

[30] Dao, Tri, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. 2022. “Hungry Hungry Hippos: Towards Language Modeling with State Space Models.” arXiv Preprint arXiv:2212.14052.

[31] Liu, Chaoyue, and Mikhail Belkin. 2018. “Accelerating Sgd with Momentum for over-Parameterized Learning.” arXiv Preprint arXiv:1810.13395.

Overview

This blog post motivates a potential marriage between artificial intelligence (AI) and physics, especially thermodynamics. Furthermore, we discuss how building at this software and hardware intersection may be pre-requisite to democratizing fast, massively energy-efficient, and reliable AI.

Check out the pre-print for more! And join our mailing list to keep up with what we’re building at this intersection.

Thinking of AI algorithms from the perspective of physics has proven fruitful, and can allow one to unify seemingly unrelated algorithms (as we discuss below). Moreover, once AI algorithms are cast in the language of physics, one can design physics-based hardware to scale up such algorithms. Accordingly, in this blog, we build up to a full-stack paradigm that we believe will be crucial for achieving ubiquitous reliability in AI at the necessary problem (data and resource) scales.

At the heart of this full-stack is Thermodynamic AI hardware — a novel computing architecture based on fundamental building blocks that are inherently stochastic — and so casting AI software and hardware as inseparable^[1] . In contrast to other paradigms like Quantum or Analog Computing, noise becomes a crucial resource for computation.

Beyond scaling up AI, we believe this new paradigm will also deepen our understanding of the connection between physics and intelligence.

Modern AI Algorithms

Modern AI has moved away from the absolute, deterministic procedures of early machine learning models. Nowadays, probability and randomness are fully embraced and utilized in AI. Some simple examples of this are avoiding overfitting by randomly dropping out neurons (i.e., dropout), and escaping local minima during training thanks to noisy gradient estimates (i.e., stochastic gradient descent). A deeper example is Bayesian neural networks^[2], where the network’s weights are sampled from a probability distribution and Bayesian inference is employed to update the distribution in the presence of data, and so deal with data noise and uncertainty in principled fashion.

Another deep example is generative modeling with diffusion models^[3]. Diffusion models add noise to data in a forward process, and then reverse the process to generate a new datapoint (see figure illustrating this for generating an image of a leaf). These models have been extremely successful not only in image generation, but also in generating molecules^[4], proteins^[5] and chemically stable materials^[6].

AI is currently booming with breakthroughs largely because of these modern AI algorithms that are inherently random. At the same time, it is clear that AI is not reaching its full potential, because of a mismatch between software and hardware. For example, sample generation rate can be relatively slow for diffusion models^[3], and Bayesian neural networks require approximations for their posterior distributions to generate samples in reasonable time^[7]. There’s no inherent reason why digital hardware is well suited for modern AI, and indeed digital hardware is handicapping these exciting algorithms at the moment.

For production AI systems, Bayesianism in particular has been stifled from evolving beyond a relative niche because of its lack of mesh with digital hardware, despite general consensus regarding its benefits towards reliability. Indeed, today’s methods for leveraging Bayesian inference typically require exceptional mathematical sophistication given the highly technical literature around approximate (practical) Bayesian computation. Next generation production-ready software may be able to significantly reduce this gap in the near-term.

Nevertheless, the next hardware paradigm should be specifically tailored to the randomness in modern AI. Specifically, we must start viewing stochasticity as a computational resource. In doing so, we could build a hardware that uses the stochastic fluctuations produced by nature, and these fluctuations would naturally drive the computations necessary in modern AI applications. Let us now dive deeper into the building blocks of computing hardware to see what kind of hardware is needed for modern AI.

Building Blocks of Current Computers

The invention of transistors led to digital computation, where the fundamental building block is the classical bit. At an abstract level, a bit can either be in the state 0 or 1. Recent decades have seen the introduction of novel building blocks that go beyond bits. See the figure below^[8] for a comparison of bits with alternative paradigms.

Allowing the state space of a bit to be a continuum between 0 and 1 leads to probabilistic bits (p-bits)^[9]. In p-bits, all convex combinations between 0 and 1 are allowed, with the coefficients being the probabilities of the p-bit holding value 0 or 1. Such p-bits are well suited to various applications such as optimization, Ising models^[10], and other randomized algorithms. Magnetic tunnel junctions provide a potential hardware architecture for p-bits^[9][11], as do Field-Programmable Gate Arrays (FPGAs)^[12].

A further extension of state space allows for complex linear combinations of 0 and 1, that is, quantum superpositions. This extension is employed in quantum computers with the building block called a quantum bit or qubit. Qubits can be measured in different bases, with the complex square of the amplitudes in the linear combination giving the outcome probabilities. Qubits can be built from atoms, ions, electron spins, or superconducting circuits. Quantum computers^[13][14] have tremendous promise for simulating quantum materials, designing pharmaceuticals, and analyzing quantum data.   

One can see that there is a trend of moving closer towards physics-based computing, with both p-bits and qubits arguably being more physics-inspired than classical bits. Indeed, additional physics-based approaches have been developed for Boltzmann machines^[15], combinatorial optimization^[16] and neural networks^[17] . In the same spirit as this trend, we introduce a new physics-inspired approach in what follows.

A New Building Block

The aforementioned building blocks are inherently static^[18]. Ideally, the state does not change over time unless it is intentionally acted upon by a gate, in these paradigms.

However, modern AI applications involve accidental time evolution, or in other words, stochasticity. This raises the question of whether we can construct a building block whose state randomly fluctuates over time. This would be useful for naturally simulating the fluctuations in diffusion models, Bayesian inference, and other algorithms. 

The key is to introduce a new axis when plotting the state space: time. Let us define a stochastic bit (s-bit) as a bit whose state stochastically evolves over time according to a continuous time Markov chain (CTMC). CTMC is a technical term for what is essentially a random jumping process in continuous time. We can then illustrate an s-bit by plotting the state versus time, as shown in the figure below.

However, there is no reason the fundamental building block needs to be a bit. In fact, most modern AI algorithms employ continuous spaces. For example, the weights in a neural network are continuous, and feature values of data are often continuous. This motivates our proposal of a continuous, stochastic building block. 

Let us define a stochastic unit (s-unit) in an abstract sense as a real, continuous variable that undergoes Brownian motion, also known as a Wiener process in mathematics. The figure above shows how the state of an s-unit could evolve over time. We say that s-units are the fundamental building blocks of Thermodynamic AI hardware, which is the term we broadly use for this new hardware paradigm^[19]. 

Ultimately this involves a shift in perspective. Certain computing paradigms, such as quantum and analog computing, view random noise as a nuisance. Noise is currently the biggest roadblock to realizing ubiquitous commercial impact for quantum computing. On the other hand, Thermodynamic AI views noise as an essential ingredient of its operation. Stochasticity is a valuable commodity^[20], the key resource for Thermodynamic AI.

Physical Realization of Stochastic Units

An s-unit represents a continuous stochastic variable whose dynamics are governed by diffusion, drift, or drag (akin to a Brownian particle). At the heart of any physical implementation of such a variable will be a source of stochasticity. A natural starting point for implementing Thermodynamic AI hardware is analog electrical circuits^[21], as these circuits have inherent fluctuations that could be harnessed for computation. 

The most ubiquitous source of noise in electrical circuits is thermal noise, also called Johnson-Nyquist noise^[22]. This noise comes from the random thermal agitation of the charge carriers in a conductor, resulting in fluctuations in voltage or current inside the conductor.

The s-unit can be represented through the dynamics of any degree of freedom of an electrical circuit (voltage, current, charge, etc.). If we choose the voltage on a particular node in a circuit as our degree of freedom of choice, we can implement an s-unit using a noisy resistor, illustrated in the following figure.

Here we modeled the noisy resistor as a combination of a stochastic voltage source (representing the thermal voltage fluctuations inside the noisy resistor), 𝛿v, an ideal noiseless resistor of resistance, R,  and a capacitor of capacitance, C. 

The stochastic dynamics of the s-unit in this circuit are constrained by the effect of the resistor and capacitor in the form of a drift (analogous to a force pushing a particle undergoing a random walk in a particular direction). In order for the s-units to be more expressive for computation, their inherent stochastic dynamics must be constrained to the properties of an algorithm. In general, one can add other electrical components, such as inductors or non-linear elements, to further constrain the evolution of the s-unit.

We note that one will need a means of coupling a system of s-units to generate correlations between them. There are a variety of options for this purpose, such as capacitive or resistive bridges that connect the s-units. Moreover, the bridges allow one to account for the geometry of the AI problem, and choosing a particular geometry for the couplings amounts to choosing an inductive bias for the AI model. In this sense, the couplings between s-units allow one to tailor the system to different problem geometries.

Unifying Algorithms via Thermodynamics

Let us now see why s-units are useful, but first we need to understand the common features that unify the AI algorithms of interest.

In physics, unifications are very powerful. For example, in the 19th century, James Clerk Maxwell unified three phenomena: electricity, magnetism, and light, under the same umbrella. Maxwell’s equations for electromagnetism showed that these seemingly unrelated phenomena were different aspects of one underlying physical force. 

Our goal is to unify modern AI algorithms. The key to unifying these algorithms is recognizing similarities between them. Namely, many of them: (1) use stochasticity and (2) are inspired by classical physics. The branch of classical physics that involves stochasticity is known as Thermodynamics. Hence, it is natural to propose Thermodynamics as a field that could unify modern AI algorithms.

Since “modern” is a vague term, let us use the term Thermodynamic AI algorithms to describe the type of algorithms that we are interested in. Here we mention a few examples of Thermodynamic AI algorithms, although this is not a complete list:

1. Generative diffusion models 

2. Hamiltonian Monte Carlo 

3. Simulated Annealing 

Note that each of these algorithms is physics-inspired. Generative diffusion models are inspired by classical diffusion, such as the Brownian motion of particles in a liquid or gas. Hamiltonian Monte Carlo^[23], which is an algorithm sampling from a distribution, is inspired by Hamilton’s equation for position and momentum in classical mechanics. Simulated annealing is an optimization algorithm that is inspired by annealing in materials science, whereby heating and then cooling a material allows the atoms to rearrange so that the system progresses towards its equilibrium state.

Through careful consideration, one can formulate a mathematical framework that encompasses all of the aforementioned algorithms as special cases. Just as Maxwell proposed a system of differential equations to unify electromagnetism, we propose the following system of differential equations as being the fundamental equations of Thermodynamic AI algorithms:

For those who are versed in classical physics, these equations should look familiar. Here, p corresponds to the momentum vector, x is the position vector, f is the force vector, U is the potential energy function, and M is the mass matrix. The matrices B and D are hyper-parameters, e.g., D can be viewed as a diffusion matrix. The term dw represents Brownian motion (also known as a Wiener process), and hence this is the source of stochasticity. Note that these equations are essentially Newton’s laws of motion, with the addition of a friction term and a diffusion term. Mathematically, the first equation is a stochastic differential equation, the second is an ordinary differential equation, and the third is a partial differential equation.

We argue that the aforementioned algorithms are special cases of the above set of equations. In practice, essentially all of the application-specific information about the problem is encoded in the potential energy function U. In simulated annealing, U corresponds to the loss function that one would like to optimize. In Hamiltonian Monte Carlo, U is the logarithm of the probability distribution that one would like to sample from. In Diffusion Models, U is the logarithm of the noise-perturbed probability distribution (i.e., the data distribution with noise added).

The implications of this unification are significant. First, this unification implies that a single software view can be used to benchmark and implement all Thermodynamic AI algorithms. Second, and more importantly, it implies that a single hardware platform could be used to accelerate all such algorithms. Third, it implies that this hardware can be based on classical physics. Again, we call the latter: Thermodynamic AI hardware.

James Clerk Maxwell’s Demon

Let us give some intuition for how one can build this hardware based on classical physics. As one can imagine, the momentum vector p corresponds to the state variable of the stochastic units (s-units) described above, since p evolves according to a stochastic equation. But the s-units are only half the story. 

We still need to introduce a system to evolve the position x in time, a system to compute the force f, and a system to apply the force as a drift term in the differential equation for p. It turns out that there is a system that can accomplish all of that, and it was introduced in the field of Thermodynamics once again by Maxwell. Maxwell considered an experiment where an intelligent observer monitors the gas particles in a box with two chambers and selectively opens the door for only fast moving particles, resulting in a separation of fast and slow particles over time (see figure). This reduces the entropy of the gas system over time, although it does not violate the 2nd law of thermodynamics since entropy is created elsewhere. The intelligent observer is dubbed Maxwell’s demon, and this demon has been experimentally realized in various physical systems^[24][25], including electrical circuits^[26].

The demon has the goal of separating the particles, and choosing this goal corresponds to choosing the application-specific potential energy function U. The demon computes the force f required and then applies it to the gas. One can therefore see that the device that we need in Thermodynamic AI is a Maxwell’s demon. 

Going back to Thermodynamic AI algorithms, we can provide a simple, conceptual definition of such algorithms as one consisting of at least two subroutines:

    (1) A subroutine in which a stochastic differential equation (SDE) is evolved over time.

    (2) A subroutine in which a Maxwell's demon observes the state variable in the SDE and applies a drift term in response.

Hence, in addition to a physical device consisting of s-units, Thermodynamic AI hardware must also include a physical device corresponding to a Maxwell’s demon.

Computational Advantages Through Physics

Let us highlight three key subroutines where Thermodynamic AI hardware is expected to provide a speedup or advantage over standard digital computers for Thermodynamic AI algorithms:

1. Generating stochasticity: Simulating random noise on a digital device is somewhat unnatural; computers expend a significant amount of energy to prepare and keep deterministic states, which become the object of all computations, including those with random subroutines. In the language of physics, the computational state of a digital computer has zero-entropy throughout any program, even for the case of generating pseudo-random noise.  In contrast,  Thermodynamic AI hardware is driven by noisy physical systems that produce states with non-zero entropy. The s-unit system provides stochastic noise on demand, while digital hardware would need to expend computational effort to produce such noise. Given that Modern AI algorithms are hungry for stochasticity and entropy, Thermodynamic AI hardware can provide an advantage here. 

2. Integrating dynamics: Stochastic differential equations are generally difficult to integrate digitally, as they can be numerically unstable especially in high dimensions, and they require careful scheduling of time step choices. With s-unit systems, on the other hand, there is no need to choose a time step and no instability, since the integration happens via physical evolution. In addition, digital approaches necessarily involve large matrix multiplications at every time step (e.g., computing BM^-1p). In contrast, the physical evolution of s-units requires no matrix multiplications. Also, digital approaches would need to simultaneously evolve the differential equations for p and x as coupled equations. However, the physical approach taken by Thermodynamic AI hardware would have p and x corresponding to conjugate variables of the same physical system. These variables naturally evolve together over time in physical systems; this is known as phase space dynamics. For example, the figure below illustrates the phase space dynamics of a damped harmonic oscillator. Thus, no extra effort is needed to couple the p and x equations in physical systems, since they are naturally coupled. 

3. Computing forces: A digital computer would need to compute the gradient of the potential energy at every time step in order to obtain the force f. In contrast, Thermodynamic AI hardware could, in principle, just measure this force instead of computing it. To elaborate, a physics-based Maxwell’s demon would essentially set up a (physical) potential energy surface U(x) for the position variable x. If we assume that x and p are conjugate variables of the same physical system, namely the s-unit system, then the Maxwell’s demon is tasked with setting up the potential energy surface for this s-unit system. But once this potential energy surface is set up, the dynamics of the s-unit system will naturally happen and the force (associated with the Maxwell’s demon) will naturally occur without any additional effort. In this sense, forces are an automatic byproduct of physical potential energy surfaces, i.e., they come for free. This is illustrated in the figure below. 

A New Full-Stack Paradigm 

This brings us to the conclusion of this blog post. As a final remark, we mention a key technological implication of the above discussion: a single, unified, full-stack paradigm can be utilized for all Thermodynamic AI algorithms. Here is a simple schematic diagram for this full-stack perspective: 

The stack is composed of a digital layer and a physical analog layer. At the top of the stack are applications and software, which will ultimately be used to program the hardware. Below the software layer is a training and optimization layer, whose job is to train the Entropy Regulator. Here we use the technical term Entropy Regulator in place of the casual term Maxwell’s demon that we used above. The Entropy Regulator is one component of the physical analog stack, and its job is to regulate the entropy of the stochastic unit system. The Entropy Regulator acts to guide the stochastic process in the right direction to solve the problem of interest. At the bottom of the stack is the stochastic unit system, which is analog hardware consisting of s-units evolving over time according to stochastic differential equations, as we describe above.

Conclusion

We hope this post stimulates discussion around Thermodynamic AI, including concepts like viewing stochasticity as a resource, and viewing AI algorithms from the lens of physics. 

Stay tuned for more about how we’re building out this future at Normal Computing — join our waitlist for the Normal Computing AI platform!

References

[1] Hinton, G. (2022). The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345.

[2] Goan, E., & Fookes, C. (2020). Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018, 45-87.

[3] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840-6851. 

[4] Corso, G., Stärk, H., Jing, B., Barzilay, R., & Jaakkola, T. (2022). Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776. 

[5] Wu, K. E., Yang, K. K., Berg, R. V. D., Zou, J. Y., Lu, A. X., & Amini, A. P. (2022). Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.15611.

[6] Xie, T., Fu, X., Ganea, O. E., Barzilay, R., & Jaakkola, T. (2021). Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197.

[7] Izmailov, P., Vikram, S., Hoffman, M. D., & Wilson, A. G. G. (2021). What are Bayesian neural network posteriors really like?. International conference on machine learning, 4629-4640.

[8] Knill, E., Laflamme R., Barnum, H. N., Dalvit, D. A., Dziarmaga, J.  J., Gubernatis, J. E., Gurvits, L., Ortiz, G., Viola, L., and Zurek, W. H. (2002). "Quantum Information Processing: A hands-on primer." Los Alamos Science, no. 27.

[9] Camsari, K. Y., Sutton, B. M., & Datta, S. (2019). P-bits for probabilistic spin logic. Applied Physics Reviews, 6(1), 011305.

[10] Chou, J., Bramhavar, S., Ghosh, S., & Herzog, W. (2019). Analog coupled oscillator based weighted Ising machine. Scientific reports, 9(1), 14786.

[11] Camsari, K. Y., Faria, R., Sutton, B. M., & Datta, S. (2017). Stochastic p-bits for Invertible Logic. Physical Review X, 7, 031014.

[12] Aadit, N. A., Grimaldi, A., Carpentieri, M., Theogarajan, L., Martinis, J. M., Finocchio, G., & Camsari, K. Y. (2022). Massively parallel probabilistic computing with sparse Ising machines. Nature Electronics, 5(7), 460-468.

[13] Preskill, J. (2012). Quantum computing and the entanglement frontier. arXiv preprint arXiv:1203.5813.

[14] Nielsen, M. A., & Chuang, I. L. (2011). Quantum Computation and Quantum Information: 10th Anniversary Edition. Cambridge University Press.

[15] Amin, M. H., Andriyash, E., Rolfe, J., Kulchytskyy, B., & Melko, R. (2018). Quantum boltzmann machine. Physical Review X, 8(2), 021050.

[16] King, A. D., Suzuki, S., Raymond, J., Zucca, A., Lanting, T., Altomare, F., ... & Amin, M. H. (2022). Coherent quantum annealing in a programmable 2,000 qubit Ising chain. Nature Physics, 18(11), 1324-1328.

[17] Wright, L. G., Onodera, T., Stein, M. M., Wang, T., Schachter, D. T., Hu, Z., & McMahon, P. L. (2022). Deep physical neural networks trained with backpropagation. Nature, 601(7894), 549-555.

[18] Some implementations of p-bits are inherently stochastic^[9][11], although in this case, the stochasticity is not the end goal but rather a means to create a stream of random bits for p-bit construction.

[19] Stochastic fluctuations naturally occur in thermodynamics, and hence this explains our coining of the term Thermodynamic AI.

[20] Mansinghka, V. K. (2009). Natively Probabilistic Computation. Ph.D. Thesis, Citeseer.

[21] We remark that digital stochastic circuits were previously considered^[20], although that differs from our analog approach.

[22] Horowitz, P., Hill, W., (2015). The art of electronics; 3rd ed.. Cambridge University Press.

[23] Neal, R. M. (2011). MCMC using Hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11), 2.

[24] Vidrighin, M. D., Dahlsten, O., Barbieri, M., Kim, M. S., Vedral, V., & Walmsley, I. A. (2016). Photonic Maxwell’s demon. Physical review letters, 116(5), 050401.

[25] Lloyd, S. (1997). Quantum-mechanical Maxwell’s demon. Physical Review A, 56(5), 3374.

[26] Kish LB & Granqvist C.-G., (2012). Electrical Maxwell Demon and Szilard Engine Utilizing Johnson Noise, Measurement, Logic and Control. PLoS ONE 7(10): e46800.

Subscribe now