By 1988, Wei Zhang et al. had discussed fast optical implementations of convolutional neural networks for alphabet recognition.[12][13]
In the 1990s, there were also attempts to create parallel high-throughput systems for workstations aimed at various applications, including neural network simulations.[14][15]
This presentation covers a past attempt at neural net accelerators, notes the similarity to the modern SLI GPGPU processor setup, and argues that general purpose vector accelerators are the way forward (in relation to RISC-V hwacha project. Argues that NN's are just dense and sparse matrices, one of several recurring algorithms)[16]
FPGA-based accelerators were also first explored in the 1990s for both inference and training.[17][18]
In 2014, Chen et al. proposed DianNao (Chinese for "electric brain"),[19] to accelerate deep neural networks especially. DianNao provides the 452 Gop/s peak performance (of key operations in deep neural networks) only in a small footprint of 3.02 mm2 and 485 mW. Later, the successors (DaDianNao,[20] ShiDianNao,[21] PuDianNao[22]) are proposed by the same group, forming the DianNao Family[23]
Heterogeneous computing incorporates many specialized processors in a single system, or a single chip, each optimized for a specific type of task. Architectures such as the
Cell microprocessor[26] have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic,
dataflow architecture, and prioritizing throughput over latency. The Cell microprocessor has been applied to a number of tasks[27][28][29] including AI.[30][31][32]
In the 2000s,
CPUs also gained increasingly wide
SIMD units, driven by video and gaming workloads; as well as support for packed low-precision
data types.[33] Due to the increasing performance of CPUs, they are also used for running AI workloads. CPUs are superior for
DNNs with small or medium-scale parallelism, for sparse DNNs and in low-batch-size scenarios.
Use of GPU
Graphics processing units or GPUs are specialized hardware for the manipulation of images and calculation of local image properties. The mathematical basis of neural networks and
image manipulation are similar,
embarrassingly parallel tasks involving matrices, leading GPUs to become increasingly used for machine learning tasks.[34][35]
In 2012, Alex Krizhevsky adopted two GPUs to train a deep learning network, i.e., AlexNet,[36] which won the champion of the ISLVRC-2012 competition. During the 2010s, GPU manufacturers such as
Nvidia added deep learning related features in both hardware (e.g., INT8 operators) and software (e.g., cuDNN Library).
Over the 2010s GPUs continued to evolve in a direction to facilitate deep learning, both for training and inference in devices such as
self-driving cars.[37][38] GPU developers such as Nvidia
NVLink are developing additional connective capability for the kind of dataflow workloads AI benefits from. As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated
neural network-
specific hardware to further accelerate these tasks.[39][40] Tensor
cores are intended to speed up the training of neural networks.[40]
GPUs continue to be used in large-scale AI applications. For example,
Summit, a supercomputer from IBM for
Oak Ridge National Laboratory,[41] contains 27,648
Nvidia Tesla V100 cards, which can be used to accelerate deep learning algorithms.
In-memory computing with analog resistive memories
In 2019, researchers from Politecnico di Milano found a way to solve systems of linear equations in a few tens of nanoseconds via a single operation. Their algorithm is based on
in-memory computing with analog resistive memories which performs with high efficiencies of time and energy, via conducting
matrix–vector multiplication in one step using Ohm's law and Kirchhoff's law. The researchers showed that a feedback circuit with cross-point resistive memories can solve algebraic problems such as systems of linear equations, matrix eigenvectors, and differential equations in just one step. Such an approach improves computational times drastically in comparison with digital algorithms.[68]
Atomically thin semiconductors
In 2020, Marega et al. published experiments with a large-area active channel material for developing logic-in-memory devices and circuits based on
floating-gatefield-effect transistors (FGFETs).[69] Such atomically thin
semiconductors are considered promising for energy-efficient
machine learning applications, where the same basic device structure is used for both logic operations and data storage. The authors used two-dimensional materials such as semiconducting
molybdenum disulphide to precisely tune FGFETs as building blocks in which logic operations can be performed with the memory elements. [69]
Integrated photonic tensor core
In 1988, Wei Zhang et al. discussed fast optical implementations of
convolutional neural networks for alphabet recognition.[12][13]
In 2021, J. Feldmann et al. proposed an integrated
photonichardware accelerator for parallel convolutional processing.[70] The authors identify two key advantages of integrated photonics over its electronic counterparts: (1) massively parallel data transfer through
wavelength division
multiplexing in conjunction with
frequency combs, and (2) extremely high data modulation speeds.[70] Their system can execute trillions of multiply-accumulate operations per second, indicating the potential of
integratedphotonics in data-heavy AI applications.[70] Optical processors that can also perform backpropagation for artificial neural networks have been experimentally developed.[71]
Nomenclature
As of 2016, the field is still in flux and vendors are pushing their own marketing term for what amounts to an "AI accelerator", in the hope that their designs and
APIs will become the
dominant design. There is no consensus on the boundary between these devices, nor the exact form they will take; however several examples clearly aim to fill this new space, with a fair amount of overlap in capabilities.
In the past when consumer
graphics accelerators emerged, the industry eventually adopted
Nvidia's self-assigned term, "the GPU",[72]
as the collective noun for "graphics accelerators", which had taken many forms before settling on an overall
pipeline implementing a model presented by
Direct3D.
All models of Intel
Meteor Lake processors have a Versatile Processor Unit (VPU) built-in for accelerating
inference for computer vision and deep learning.[73]
Deep Learning Processors (DLP)
Inspired from the pioneer work of DianNao Family, many DLPs are proposed in both academia and industry with design optimized to leverage the features of deep neural networks for high efficiency. Only at ISCA 2016, three sessions, 15% (!) of the accepted papers, are all architecture designs about deep learning. Such efforts include Eyeriss (MIT),[74] EIE (Stanford),[75] Minerva (Harvard),[76] Stripes (University of Toronto) in academia,[77] TPU (Google),[78] and MLU (
Cambricon) in industry.[79] We listed several representative works in Table 1.
The major components of DLPs architecture usually include a computation component, the on-chip memory hierarchy, and the control logic that manages the data communication and computing flows.
Regarding the computation component, as most operations in deep learning can be aggregated into vector operations, the most common ways for building computation components in digital DLPs are the
MAC-based (multiplier-accumulation) organization, either with vector MACs[19][20][22] or scalar MACs.[78][21][74] Rather than
SIMD or
SIMT in general processing devices, deep learning domain-specific parallelism is better explored on these MAC-based organizations. Regarding the memory hierarchy, as deep learning algorithms require high bandwidth to provide the computation component with sufficient data, DLPs usually employ a relatively larger size (tens of kilobytes or several megabytes) on-chip buffer but with dedicated on-chip data reuse strategy and data exchange strategy to alleviate the burden for memory bandwidth. For example, DianNao, 16 16-in vector MAC, requires 16 × 16 × 2 = 512 16-bit data, i.e., almost 1024 GB/s bandwidth requirements between computation components and buffers. With on-chip reuse, such bandwidth requirements are reduced drastically.[19] Instead of the widely used cache in general processing devices, DLPs always use scratchpad memory as it could provide higher data reuse opportunities by leveraging the relatively regular data access pattern in deep learning algorithms. Regarding the control logic, as the deep learning algorithms keep evolving at a dramatic speed, DLPs start to leverage dedicated ISA (instruction set architecture) to support the deep learning domain flexibly. At first, DianNao used a VLIW-style instruction set where each instruction could finish a layer in a DNN. Cambricon[84] introduces the first deep learning domain-specific ISA, which could support more than ten different deep learning algorithms. TPU also reveals five key instructions from the CISC-style ISA.
Hybrid DLPs
Hybrid DLPs emerge for DNN inference and training acceleration because of their high efficiency. Processing-in-memory (PIM) architectures are one most important type of hybrid DLP. The key design concept of PIM is to bridge the gap between computing and memory, with the following manners: 1) Moving computation components into memory cells, controllers, or memory chips to alleviate the memory wall issue.[81][85][86] Such architectures significantly shorten data paths and leverage much higher internal bandwidth, hence resulting in attractive performance improvement. 2) Build high efficient DNN engines by adopting computational devices. In 2013, HP Lab demonstrated the astonishing capability of adopting ReRAM crossbar structure for computing.[87] Inspiring by this work, tremendous work are proposed to explore the new architecture and system design based on ReRAM,[80][88][89][81] phase change memory,[85][90][91] etc.
Benchmarks
Benchmarks such as MLPerf and others may be used to evaluate the performance of AI accelerators.[92] Table 2 lists several typical benchmarks for AI accelerators.
^
abZhang, Wei (1988). "Shift-invariant pattern recognition neural network and its optical architecture". Proceedings of Annual Conference of the Japan Society of Applied Physics.
^
abGschwind, M.; Salapura, V.; Maischberger, O. (1996). "A Generic Building Block for Hopfield Neural Networks with On-Chip Learning". 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96. pp. 49–52.
doi:
10.1109/ISCAS.1996.598474.
ISBN0-7803-3073-0.
S2CID17630664.
^Michael Feldman (May 23, 2018).
"Intel Lays Out New Roadmap for AI Portfolio". TOP500 Supercomputer Sites. Retrieved May 23, 2018. Intel plans to support this format across all their AI products, including the Xeon and FPGA lines
^Lucian Armasu (May 23, 2018).
"Intel To Launch Spring Crest, Its First Neural Network Processor, In 2019". Tom's Hardware. Retrieved May 23, 2018. Intel said that the NNP-L1000 would also support bfloat16, a numerical format that's being adopted by all the ML industry players for neural networks. The company will also support bfloat16 in its FPGAs, Xeons, and other ML products. The Nervana NNP-L1000 is scheduled for release in 2019.
^Elmar Haußmann (April 26, 2018).
"Comparing Google's TPUv2 against Nvidia's V100 on ResNet-50". RiseML Blog. Archived from
the original on April 26, 2018. Retrieved May 23, 2018. For the Cloud TPU, Google recommended we use the bfloat16 implementation from the official TPU repository with TensorFlow 1.7.0. Both the TPU and GPU implementations make use of mixed-precision computation on the respective architecture and store most tensors with half-precision.
^Joshua V. Dillon; Ian Langmore; Dustin Tran; Eugene Brevdo; Srinivas Vasudevan; Dave Moore; Brian Patton; Alex Alemi; Matt Hoffman; Rif A. Saurous (November 28, 2017). TensorFlow Distributions (Report).
arXiv:1711.10604.
Bibcode:
2017arXiv171110604D. Accessed May 23, 2018. All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts
^Ji, Yu Zhang, Youyang Xie, Xinfeng Li, Shuangchen Wang, Peiqi Hu, Xing Zhang, Youhui Xie, Yuan (January 27, 2019). FPSA: A Full System Stack Solution for Reconfigurable ReRAM-based NN Accelerator Architecture.
OCLC1106329050.{{
cite book}}: CS1 maint: multiple names: authors list (
link)
^Nandakumar, S. R.; Boybat, Irem; Joshi, Vinay; Piveteau, Christophe; Le Gallo, Manuel; Rajendran, Bipin; Sebastian, Abu; Eleftheriou, Evangelos (November 2019). "Phase-Change Memory Models for Deep Learning Training and Inference". 2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS). IEEE. pp. 727–730.
doi:
10.1109/icecs46596.2019.8964852.
ISBN978-1-7281-0996-1.
S2CID210930121.