31
2024
-
07
Optica | Optical pruning enables reliable optical computing
Author:
Reading:
In recent years, the development of integrated photonic neural network chip is gradually scale and mature. Taking advantage of the unique advantages of light, the integrated photonic neural network chip has the advantages of high bandwidth, low latency and low power consumption. However, the integrated photonic neural network chip itself is very sensitive to external disturbances, thermal crosstalk and various device imperfections, which seriously restricts the accuracy and reliability of the photonic computing chip. Usually we need to use complex control methods to ensure the stable operation of the chip, but this will make the hardware system very complex, which is not conducive to the large-scale application of integrated photonic neural networks.
Recently, researchers from the Chinese University of Hong Kong, Princeton University, Queen's University and Beijing University of Posts and Telecommunications have jointly proposed a new training method for integrated photonic neural networks, which realizes the stability of the integrated photonic neural network chip without increasing hardware overhead., Accurate, and low-power operation. The research work is based on"Control-free and efficient integrated photonic neural networks via hardware-aware training and pruningThe title is published inOpticIn the magazine. The Chinese University of Hong KongProfessor Wong ChaoranFor the corresponding author of this article. The Chinese University of Hong KongPhD student Xu TengjiPrinceton UniversityDr. Zhang WeipengandDoctor Zhang JiaweiCo-first author of this article. Co-authors include The Chinese University of Hong KongResearch Assistant Zeyu Luo,Doctoral Studies Zodiac Chia Chia Rung,Wang BenshanandLuo MingchengBeijing University of Posts and TelecommunicationsProfessor XU Xing-yuanQueen's UniversityBhavin J. Shastri教授and Princeton UniversityPaul R. Prucnal教授.
During the operation of the photonic neural network chip, the linear connection weight of the electrical neural network is deployed on a specific optical device, and different weight connections are realized by electrically adjusting the specific optical device. However, optical devices such as microring resonators are highly sensitive to the environment, which leads to a large difference between the reasoning effect of photonic chips and the theoretical results. The traditional method to solve this problem requires a lot of additional hardware overhead, such as high power consumption temperature control device to stabilize each micro-ring resonator.
At the same time, the in-memory calculation optical neural network chip based on phase-change material will also face many difficulties, such as limited weight resolution, material degradation, randomness in the crystallization process and temperature fluctuations in the crystal state switching in the experiment. These problems can cause the neural network's recognition accuracy to drop from 99.0 percent to 63.2 percent at small temperature jitters (<5K). The traditional solution to these problems is still to use additional dynamic stabilization devices to stabilize the entire system, but this also requires a lot of additional hardware investment.
In view of this, we use the noise robust region and noise sensitive region of the micro-ring resonator itself, and transfer the weight of the neural network from the noise sensitive region to the noise robust region by adding a regularization term in the process of training the neural network, so as to improve the robustness and accuracy of the photon computing chip. At the same time, the noise robust region also has the advantages of low energy consumption and high signal-to-noise ratio, so the proposed training method can also reduce the crosstalk between the micro-ring resonators and improve the signal-to-noise ratio of the system operation. Compared with the traditional method, the advantage of our method is that this method significantly improves the reliability and accuracy of the photonic computing chip without increasing the hardware overhead.

Figure 1: The performance of the micro-ring weight library in neural network hardware reasoning, the error performance of the micro-ring runtime, and the optimization results of the optical pruning method. (a-i) convert convolutional layers and fully connected layers into matrix multiplication. (a-ii) The matrix multiplication is implemented on the micro-ring weight library. (b-i) Change the spectrum of a single microring resonator by thermal tuning. (b-ii) Plot of tuning power and weight error as a function of weight value. (b-iii) The signal transmission diagram of the microring resonator when the weights are 0 and 1. (b-iv) A plot of the signal-to-noise ratio as a function of the weight value. (c-i) Statistical histogram of the neural network weight distribution. (c-ii) Statistical histogram of micro-ring weight library tuning power consumption distribution. (c-iii) Statistical histogram of the weight error distribution.
Photonics neural networks can be implemented by high-speed optical devices densely connected by parallel waveguides. Among them, the synaptic weights in the convolutional layer and the fully connected layer can be equivalently converted into matrix multiplication. The matrix operation can be achieved by incoherent summation of light intensity or coherent superposition of light interference. Currently widely used incoherent systems include microrings weight libraries, crossbar microrings arrays, and crossbar phase change material arrays. These systems achieve different light intensity transmissivity by tuning the phase shifter, thus achieving different weights. The different wavelengths of light will eventually be added by a single-ended detector or a balanced detector. Compared with coherent methods such as Mach-Zehnder Interferometer (MZI) networks, incoherent optical neural networks have the advantages of tuning efficiency, direct weight distribution, small size and high computational density. However, the accuracy of weight deployment may be affected by environmental interference, crosstalk, limited weight resolution, and device degradation. Although these problems can be solved by retraining the photonic neural network or using control circuits to lock the weight states, these methods greatly increase the hardware complexity.
Adding a regularization term to the training process increases the training period, but from the perspective of achieving a more efficient and reliable photonic neural network, the benefits far outweigh the initial training cost. This is because training is a one-time event, and the neural network hardware deployed after training runs continuously. In addition, training is usually only performed on a single high-performance computer or computer cluster, and the trained parameters are deployed to many photonic neural networks for inference. Therefore, it is most important to prioritize the reliability and power consumption of the parameters deployed to the photonic neural network, even if the optical pruning requires additional training burden. This method of adapting edge devices to power consumption and reliability requirements through pruning has been widely accepted in the field of computers and electronic hardware.
Research Results
1. Realize the stable operation of photonic computing chip without temperature control
We verify the proposed method from two angles: first, verify that the method can effectively reduce the weight deployment error in the absence of temperature control, and then deploy the electrical network to the optical hardware for reasoning to verify whether the effect of environmental temperature drift will significantly improve the accuracy in the absence of temperature control.
In the first step of the verification of weight errors, we chose a two-layer convolutional neural network architecture for recognizing the handwritten digit set MNIST. When the optical pruning method is not used, the theoretical accuracy of the neural network is 96.5, and the weight is concentrated near 0. When these trained weights are deployed to the micro-ring weight library, the error is significant because the weights are concentrated in the noise-sensitive area. Then, we use the optical pruning method to transfer as much weight as possible to the noise robust region without reducing the recognition accuracy, and finally about 55.4 percent of the weight is transferred to the region. Similarly, when these weights are deployed to the micro-ring weight library, the weight error is significantly reduced, and the accuracy can be improved up to 4 bits.

Figure 2: Micrograph of the microring weight library, experimental weight error and correlation of weight error between adjacent microring. (a-i) 4 x 1 micro-ring weight library for point multiplication. The light of different wavelength modulation is combined by wavelength division multiplexer, weighted by a series of micro-rings, and finally summed in the balanced detector. (a-ii) Spectra of the microring weight library. (a-iii) The experimentally measured tuning curve. (b-i) the change of the weight distribution of the micro-ring during the optimization process. (b-ii) Weighted error distribution without pruning. (b-iii) Weighted error distribution with pruning. (c) Correlation of weight errors between adjacent microrings.
The second step is to verify the electrical network on the optical hardware to determine whether the accuracy is significantly improved. We convert the convolution operation into matrix multiplication, and inject the information encoded on the light into the micro-ring weight library for matrix multiplication, and then deploy the entire network on the micro-ring array for hardware reasoning. The experimental results show that in the case of no optical pruning, too large weight error will lead to the accuracy rate dropped to 67%. In contrast, after using optical pruning, the recognition accuracy rate is improved to 95%, which is close to the theoretical value. In addition, we also verify the effectiveness of this method in the LeNet-5 and ResNet-18 of deep neural networks through simulation. About 92.1 percent of the weight can be transferred to the noise-robust region when the MNIST handwritten digit recognition task is LeNet-5. With the increase of the network, about 98.9% of the weight can be transferred to the noise robust region when using ResNet-18 for CIFAR-10 recognition tasks. The more weight is transferred to the noise robust region, the more obvious the error is reduced. At the same time, the tuning power of the micro-ring will also decrease, and this downward trend becomes more significant with the increase of the network scale.

Figure 3: Comparison of hardware inference accuracy of different sizes of neural networks under the micro-ring weight library. (a-i) Recognition confusion matrix of two-layer convolutional neural network without pruning training results. (a-ii) LeNet-5 the recognition confusion matrix without pruning training results. (a-iii) ResNet-18 the recognition confusion matrix without pruning training results. (b-i) The two-layer convolutional neural network has a recognition confusion matrix with pruning training results. (b-ii) LeNet-5 a recognition confusion matrix with pruning training results. (b-iii) ResNet-18 a recognition confusion matrix with pruning training results.

Figure 4: Average values of microrings tuning power for neural networks of different sizes.
In addition, we extend this method to other types of micro-ring arrays. In these arrays, the nonlinear relationship between tuning power and weight shows that there is still sufficient optimization space to transfer the weight from the noise-sensitive region to the noise-robust region.

Figure 5: Applying the optical pruning method to other types of microrings arrays. (a-i) Schematic diagram of single-ended detection micro-ring array. (a-ii) Microrings 'all-pass' port transmittance tuning curve. (a-iii) Comparison of the distribution of the absolute value of the weight in the case of 50pm wavelength shift. (b-i) Schematic of the vertical and horizontal micro-ring array. (b-ii) Microrings 'download' port transmittance tuning curve. (b-iii) Absolute value distribution of weights in case of wavelength shift of 150pm.
2. Solve the instability of phase change materials.
We also show that our method can solve the problem of chip calculation accuracy degradation caused by the instability of phase change materials. Optical neural networks based on phase change materials achieve tunable weights by thermally adjusting the ratio between crystalline and amorphous states. In the crystalline state, most of the input light is absorbed, and the corresponding weight value is "0". In the amorphous state, most of the light is transmitted, corresponding to a weight value of "1". By heating the phase change material to the melting point temperature TmAfter rapid cooling and then heating to crystallization temperature Tcand melting point temperature TmThe weight of the phase change material can be adjusted between specific temperature values. This allows the phase change material to be partially crystallized to a specific optical state, achieving a specific weight between 0 and 1. However, the instability of phase change materials, especially temperature fluctuations, can affect the crystallization ratio, introduce weight errors, and cause limited weight resolution. Randomness in the material degradation and crystallization process can also cause problems. At small temperature jitters (<5K), the neural network's recognition accuracy dropped from 99.0 percent to 63.2 percent.
In order to solve this problem, we first analyze the stable region of photonic neural networks based on phase change materials. It can be observed from Figure 6(c) and (d) that the pure crystalline and amorphous phase change material states are not sensitive to temperature fluctuations. Therefore, by adding the corresponding regularization term to the training loss function, we achieve the transfer of weights to the noise-robust crystalline and amorphous states.


By transferring the weights to the noise-robust region, the accuracy of the weight configuration of the neural network is improved from 3.99 bits to 6.40 bits. In the case of slight temperature jitter (<5K), the recognition accuracy of the neural network also increased from 63.2 percent to 98.5 percent, which is close to the theoretical limit.

Figure 6: Applying the optical pruning method to a phase change material array. (a) Schematic diagram of the phase change material array. (B) Schematic diagram of the process of adjusting the weight of the phase change material. (c) A diagram showing the relationship between the crystallization ratio and the heating temperature. The gray dashed line represents the 8 weight values that can be clearly resolved at 5K temperature jitter. (D) Diagram of the relationship between weight and heating temperature. (E) Comparison of the absolute value distribution of weights under slight temperature jitter (<5K).
Summary and Prospect
In summary, this paper improves the robustness of the optical neural network from the perspective of the combination of hardware and software by intentionally transferring the weight of the photonic neural network to the noise robust area during the training process. The improved robustness allows the photonic neural network chip to eliminate the need for complex control algorithms or additional hardware. This method provides a new idea for large-scale, stable and low-power operation of optical neural networks. In addition, the method proposed in this paper to transfer the physical neural network parameters from the noise-sensitive region to the noise-robust region can also be applied to other physical neural networks. From the system point of view, the weight of the micro-ring is transferred to the noise robust region, which can also enhance the signal strength, thus stimulating stronger nonlinearity in the optical nonlinear function and reducing the optical power requirements of the system.
Chongqing Chongfan Technology Co., Ltd. Optical platform film thickness meter OCT optical coherence tomography LabVIEW control