Fig. 1: A bioinspired in-materia analog photoelectronic reservoir computing (Alpho-RC) system.<\/b> a<\/b> Conceptual diagram of dynamic vision processing in human visual system. The walking man and eye are reproduced with permission from Pixabay. b<\/b> Schematic diagram of calculation processes in Alpho-RC system. c<\/b> Photograph of EDL-coupled IGZO photoelectronic transistor array and schematic diagram of IGZO transistor structure. d<\/b> The response and relaxation behaviors of EDL-coupled IGZO photoelectronic transistor in response to a light pulse (38\u2009ms, 2.95 nW\u00b7\u03bcm\u20132) under different gate bias conditions (\u22120.4, \u22120.2, and 0\u2009V, respectively). e<\/b> A micrograph image of a 32\u2009\u00d7\u200932 1T1R array (scale bar: 200\u2009\u03bcm). f<\/b> 100 cycles of I-V sweeps of a Pt\/Ta\/TaOx\/Pt memristor in the array.<\/p>\n Figure\u00a01b<\/a> shows the schematic diagram of such Alpho-RC system. The data modes of human action are mainly divided into visual modes (RGB, Depth, Skeleton and so on) and non-visual modes (Inertial acceleration data, Wireless-transmission signal and so on)18<\/a>,19<\/a>,20<\/a>,21<\/a>,22<\/a>. Among them, the visual mode is more in line with human intuitive feelings. Compared with other visual modalities, a 3D skeleton frame as a topological form of joints and skeletons abstracts the human body into a three-dimensional coordinate space, which tends to be more robust in complex environments. Therefore, 3D skeleton-based human action frames are selected as visual input in the system.<\/p>\n The core elements of spike encoding scheme are neurons with GRF without additional filtering process. In biology, stimuli carried essential features are selectively responded by neurons with overlapped and graded receptive fields, and the varied levels of features thus can be extracted in different neural pathways23<\/a>. Stimulus can significantly fire at the neurons with proper receptive field, and might be silence in other neurons. Such biological encoding mechanisms enable only a small size of data being processed by the central nervous system. In contrast, running machine learning algorithms in a digit system always requires complex feature extraction steps, which lead to a huge computational burden24<\/a>. In Alpho-RC system, several GRF neurons are defined as one population encoder, which is corresponding to population transistor consisting of electric-double-layer (EDL) coupled IGZO photoelectronic transistors. EDL coupled IGZO transistors have been demonstrated with photoelectronic synaptic plasticity by gate voltage tunability based on persistent photoconductivity (PPC) effects and proton relaxation dynamics25<\/a>,26<\/a>,27<\/a>. Figure\u00a01c<\/a> shows the photograph of photoelectronic transistor array and schematic diagram of IGZO transistor structure. Details on transistor array fabrication, basic properties and hardware operation system can be found in \u201cMethods\u201d section and Supplementary Figs.\u00a01<\/a>\u20139<\/a>. The response and relaxation behaviors of the EDL-coupled IGZO photoelectronic transistor in response to a light pulse (38\u2009ms, 2.95 nW\u00b7\u03bcm\u20132) under different gate bias conditions (\u20130.4, \u20130.2, and 0\u2009V, respectively) has shown in Fig.\u00a01d<\/a>. The electrons generated by PPC effect would increase the channel conductance transiently. When a light stimulation ends, accumulated electrons would recombine gradually with the triggered oxygen vacancies within a certain time. As a consequence, the channel conductance gradually decreases, exhibiting relaxation characteristics. A negative gate voltage would decrease the channel conductance through the EDL coupling, leading to a decreased decay time. The decrement is positively related to the absolute value of the negative gate voltage (detailed information can be found in Supplementary Note\u00a01<\/a>). By applying varied gate biases that are corresponded to GRF neurons with different distribution centers, current states of the transistor are tuned in response to the encoding pulses from GRF neurons, mapping the visual input information to the high-dimensional space in turn.<\/p>\n Physical reservoir computing (PRC) based on material intrinsic dynamics has been demonstrated with excellent time signal processing capabilities, which is selected as the calculation method in Alpho-RC system28<\/a>,29<\/a>,30<\/a>. Obtained response currents by population transistors are used as reservoir states for training output weights. Compared with the processing path by single device in traditional PRC systems, such receptive field-enhanced population encoding mechanism increases the information processing capacity by providing multiple parallel information processing ports. Meanwhile, no additional feature extraction is required to apply on skeleton sequences, through such bioinspired in-materia reservoir computing framework. However, the feature extraction is still required for skeleton frames in vast number of reported algorithms. For example, while facing the skeleton-based action recognition tasks, feature set including spatial-domain-feature (relative position, distances between joints, distances between joints and lines) and temporal-domain-feature (joint distances map, joint trajectories map) needs to be extracted prior to the training of neural networks31<\/a>.<\/p>\n The output layer was also implemented in a 32\u2009\u00d7\u200932 1T1R array for fully hardware implementation of the bioinspired in-materia reservoir computing system, as shown in Fig.\u00a01e<\/a> (the detailed device structure and fabrication can be found in Supplementary Fig.\u00a010<\/a> and \u201cMethods\u201d section). TaOx-based memristors were integrated on top of a foundry-made complementary-metal-oxide-semiconductor (CMOS) chip with silicon-based selecting transistors and fan-outs (the detailed characterizations and the setup of hardware operation system can be found in Supplementary Figs.\u00a011<\/a>\u201314<\/a>). The memristors demonstrated good switching uniformity and stability for reliable programming of offline-trained weights for human action processing (Fig.\u00a01f<\/a>). Our analog photoelectronic reservoir computing system is built based on the IGZO transistors for reservoir states collection and the TaOx-based memristors for matrix operations.<\/p>\n The bioinspired in-materia reservoir computing framework<\/p>\n The dimensionality enhancement process of the photoelectronic reservoir in Alpho-RC system contains time multiplexing and GRF encoding. Time multiplexing with mask matrixes is a common method in PRC systems, which solves the difficulty of interconnecting physical device nodes based on the concept of delayed nodes30<\/a>. Figure\u00a02a<\/a> shows the calculation flow of time multiplexing on original action skeleton information. Firstly, K body joint points with coordinate information (x<\/b>i<\/b>, y<\/b>i<\/b>, z<\/b>i<\/b>) are collected from a subject, which maps the human body\u2019s action posture into a three-dimensional coordinate space. Therefore, temporal changes of the K joint coordinate positions from frame-to-frame can describe the dynamic change of an action. 3D joint coordinates of each frame are written as one-dimensional sequence vector [x<\/b>1<\/b> y<\/b>1<\/b> z<\/b>1<\/b> x<\/b>2<\/b> y<\/b>2<\/b> z<\/b>2<\/b> \u2026\u2026 x<\/b>K<\/b> y<\/b>K<\/b> z<\/b>K<\/b>] or [J<\/b>1<\/b> J<\/b>2<\/b> \u2026\u2026 J<\/b>K<\/b>] according to the order of the skeleton joints. In this case, the dynamic coordinates of a complete the action (Act, with a size of 3\u00b7K\u00d7N) can be represented by:<\/p>\n $${Act}=[ \\, \\, {{{{\\boldsymbol{J}}}}}_{{{{\\bf{11}}}}}\\,{{{{\\boldsymbol{J}}}}}_{{{{\\bf{12}}}}}\\ldots {{{{\\boldsymbol{J}}}}}_{{{{\\bf{1}}}}{{{\\boldsymbol{k}}}}};\\ldots \\ldots ;{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{n}}}}{{{\\bf{1}}}}} \\, {{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{n}}}}{{{\\bf{2}}}}}\\ldots {{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{nk}}}}};\\ldots \\ldots ;{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{N}}}}{{{\\bf{1}}}}} \\, {{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{N}}}}{{{\\bf{2}}}}}\\ldots {{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{Nk}}}}}]$$<\/p>\n \n (1)\n <\/p>\n where n (n\u2009=\u20091, 2, \u2026, N) is the frame order of action and N is the total number of frames consisting of an action. The dynamic change of the 3D joint coordinates (\u0394Act, with a size of 3\u00b7K \u00d7 N) can be represented by the subtraction of the action matrix and the first sequence of the action:<\/p>\n $$\\varDelta {Act}=[{{{\\bf{0}}}}\\,{{{\\bf{0}}}}\\ldots {{{\\bf{0}}}};\\ldots \\ldots ;{{{\\boldsymbol{\\Delta }}}}{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{n}}}}{{{\\bf{1}}}}} \\, {{{\\boldsymbol{\\Delta }}}}{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{n}}}}{{{\\bf{2}}}}}\\ldots {{{\\boldsymbol{\\Delta }}}}{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{nk}}}}};\\ldots \\ldots ;{{{\\boldsymbol{\\Delta }}}}{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{N}}}}{{{\\bf{1}}}}} \\, {{{\\boldsymbol{\\Delta }}}}{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{N}}}}{{{\\bf{2}}}}}\\ldots {{{\\boldsymbol{\\Delta }}}}{{{{\\boldsymbol{J}}}}}_{{{{\\boldsymbol{Nk}}}}}]$$<\/p>\n \n (2)\n <\/p>\n where \u0394Jnj is the change of coordinates (\u0394Jnj=Jnj-J1j), and j is the number of joint (j\u2009=\u20091, 2,\u2026, K). Figure\u00a02b<\/a> shows examples of \u0394Actn (\u0394Actn\u2009=\u2009[\u0394J<\/b>n<\/b>1<\/b>\u0394J<\/b>n<\/b>2<\/b>\u2026 \u0394J<\/b>nK<\/b>]) corresponding to the action \u201copenarm\u201d shown in Fig.\u00a02a<\/a>. After that, \u0394Act is mapped into high dimensional space by multiplying mask matrixes. The mask matrix is a randomly generated two-dimensional matrix consisting of 1 and -1, with a size of 3\u00b7K \u00d7 M (M is the mask length). At nth frame of the action, the \u0394Actn (1 \u00d7 3\u00b7K) is multiplied by the mask matrix, generating virtual nodes (Vn\u2009=\u2009[V<\/b>n<\/b>1<\/b> V<\/b>n<\/b>2<\/b> \u2026 V<\/b>nM<\/b>]) with a size of 1\u00d7M. In this case, the \u0394Act is mapped to a higher dimensional matrix noted as PreIn matrix (1\u00d7M\u00b7N):<\/p>\n $${PreIn}=[{{{{\\boldsymbol{V}}}}}_{{{{\\bf{1}}}}}{{{{\\boldsymbol{V}}}}}_{{{{\\bf{2}}}}}\\ldots {{{{\\boldsymbol{V}}}}}_{{{{\\boldsymbol{N}}}}}]$$<\/p>\n \n (3)\n <\/p>\n Fig. 2: The Gaussian receptive field based encoding and photoelectronic reservoir computing.<\/b> a<\/b> The calculation flow of time multiplexing on original action skeleton information in photoelectronic reservoir computing. b<\/b> Examples of \u0394Actn corresponding to the action \u201copenarm\u201d in (a<\/b>). c<\/b> An example of PreIn corresponding to \u0394Actn in (b<\/b>) with a mask matrix (M\u2009=\u200910). d<\/b> The converted light pulse trains from spike trains of Neuron #1, #2, and #3 and are applied on one population transistor consists of three IGZO-based photoelectronic synaptic transistors with different modulated biases. e<\/b> The response current of transistor with gate bias of \u22120.4\u2009V to four consecutive light pulses (38\u2009ms ON and 38\u2009ms OFF). f<\/b> MC values with different numbers (1, 2, 3, and 4) of Gaussian receptive fields. The error bars represent the standard deviation of five independent tests.<\/p>\n Figure\u00a02c<\/a> shows one example of PreIn corresponding to \u0394Actn in Fig.\u00a02b<\/a> with a mask matrix (M\u2009=\u200910).<\/p>\n Then, the PreIn corresponding to a completed action is encoded to spikes through GRF neurons. The Gaussian receptive fields of the Alpho-RC system are set based on the intensity of PreIn, which is inspired from encoding process of biological systems and spiking neural networks (SNN)32<\/a>. For every input, the output through ith receptive field yields the Gaussian distribution (Gi) with a mean value of \u03bci and deviation of \u03c3i. So, the output (Gi(A)) is dependent on the distance between input intensity (A) and \u03bci:<\/p>\n $${G}_{i}(A)=\\frac{1}{\\sqrt{2\\pi {\\sigma }_{i}^{2}}}\\exp \\left(-\\frac{{\\left(A-{\\mu }_{i}\\right)}^{2}}{2{\\sigma }_{i}^{2}}\\right)$$<\/p>\n \n (4)\n <\/p>\n In this case, each input can trigger an output through all receptive fields, while the outputs are different. The GRF neuron would trigger a firing spike only if the output from the corresponding receptive field is the largest. Here, several neurons (Neuron #1, #2, \u2026 #i) corresponding to receptive fields (GRF #1, #2, \u2026 #i) are defined as one population encoder which is used for spike encoding. The \u03c3 of neurons is uniformly set to the standard value of 1, and the centers (\u03bc1,\u03bc2, \u2026\u03bci) of GRFs are set by:<\/p>\n $${{{\\mu }}}_{i}={A}_{\\min }+\\frac{{A}_{\\max }-{A}_{\\min }}{m+1}\\times i$$<\/p>\n \n (5)\n <\/p>\n where Amin and Amax represent the minimum value and the maximum value of input intensity (A), m is the number of GRF neurons. To facilitate data processing, we perform a normalization process, so that the amplitude of PreIn is limited between -1 and 1. Accordingly, Gaussian centers are set by:<\/p>\n $${{{\\mu }}}_{i}=\\frac{2}{m+1}\\times i-1$$<\/p>\n \n (6)\n <\/p>\n The normalized PreIn of a completed action is then converted into spike trains (Spike-input) through the aforementioned GRF-based population encoding scheme. The spike trains from Neuron #1, #2, \u2026 #i are converted into light pulse trains (405\u2009nm, 2.95 nW\u00b7\u03bcm\u20132) and applied on one population transistor consisting of EDL-coupled IGZO photoelectronic transistors under different gate bias conditions (as shown in Fig.\u00a02d<\/a>). Each light pulse corresponding to firing spike is set to 38\u2009ms ON, and stationary state without firing spike is set to 38\u2009ms OFF. Thus, the time step between each node of PreIn is fixed, and the triggered channel currents of population transistor at each time step are collected as reservoir states. Relaxation characteristic curves are used as the activation function. As mentioned before, the characteristics are varied in transistors with different gate biases, which increase the richness of reservoir states. Figure\u00a02e<\/a> shows the response current of transistor with gate bias of -0.4\u2009V to four consecutive light pulses (38\u2009ms ON and 38\u2009ms OFF). Supplementary Figs.\u00a015<\/a>, 16<\/a> show the device responses under different gate bias voltages to light pulse trains of different frequencies. Specific model fitting and calculation details can be found in Supplementary Fig.\u00a017<\/a>, Fig.\u00a018<\/a>, Table\u00a01<\/a> and Note 2. L population encoders are connected in parallel to build a large parallel RC system. Each population encoder is responsible for encoding different PreIn, which is generated with L different mask matrixes parallelly. Thus, L population transistors with 3\u00d7L devices in arrays are used to obtain parallel reservoir states and 3\u00d7L\u00d7M reservoir states are recorded per frame. Supplementary Video\u00a0I<\/a> shows the aforementioned spike encoding and reservoir states collection processes.<\/p>\n Finally, the reservoir states are collected for training and testing. During the training process, only the output weights (Wout) connected to the output layer are needed to be trained. The collected reservoir states are subjected to a one-step linear regression along with labels to obtain the desired weights33<\/a>. The resulting output weights are multiplied with reservoir states collected from the testing process, and the output label is obtained based on the winner-take-all method34<\/a>. Supplementary Fig.\u00a019<\/a> shows the specific procedures, and detailed calculation procedures can be found in \u201cMethods\u201d.<\/p>\n Memory capacity (MC) is a task-independent evaluation index of reservoir computing, which represents the ability of reservoir states to preserve input information previously29<\/a>,35<\/a>. We evaluated the MC metrics of the bioinspired reservoir with different numbers (1, 2, 3, and 4) of GRF neurons. During encoding, the Gaussian centers were also sequentially set according to Eq. (6<\/a>). Population encoders with different GRF neuron numbers (1, 2, 3, and 4) are corresponding to population transistor consisting of different amounts of EDL-coupled IGZO photoelectronic transistors with fixed VGS of 0\u2009V, \u20130.2\u2009V\/0\u2009V, \u20130.4\u2009V\/\u20130.2\u2009V\/0\u2009V, \u20130.8\u2009V\/\u20130.4\u2009V\/\u20130.2\u2009V\/0\u2009V, respectively. Specific calculation process of MC can be found in Supplementary Note.\u00a03<\/a>. As shown in Fig.\u00a02f<\/a>, high MC value (~6.9) is obtained by three receptive fields with filtering and encoding. This should be due to the enriched reservoir states rendered by multiple GRF neurons. When encoding number is more than three, the MC value is almost remained. However, the greater number of GRF neurons, the larger the data scale is required, which would increase the computational burden on the following steps. Therefore, three GRF neurons are fixed as one population encoder and one population transistor is fixed of three EDL-coupled IGZO photoelectronic transistors with fixed VGS of \u20130.4\u2009V, \u20130.2\u2009V, 0\u2009V, respectively in the bioinspired reservoir.<\/p>\n Human action recognition tasks based on standard datasets<\/p>\n The UTD-MHAD dataset contains 27 classes of human actions collected from 8 subjects36<\/a>. Each skeleton frame consists of dynamic coordinates of 20 body joints. Figure\u00a03a<\/a> shows examples of color images and skeleton frames corresponding to a home-made action \u201cHigh throw\u201d (left panel) and an action \u201cBasketball shoot\u201d (right panel) in UTD-MHAD dataset. Color images and skeleton frames of home-made action were collected by authors through a mobile phone camera and a Kinect camera, respectively. In the UTD-MHAD standard dataset test, we randomly selected 30 samples of each action class, and divided them into training set and testing set at a ratio of 9:1. Specific encoding processes can be found in Supplementary Fig.\u00a020<\/a>. Figure\u00a03b<\/a> shows partial time sections of reservoir states triggered by the light pulse trains and collected from the IGZO synaptic transistors. Different mask matrixes were used and the optimized performance was achieved with M\u2009=\u200930 and L\u2009=\u200930. After training and testing processes, the recognition rate was calculated by counting the recognition results of the entire data in the testing set. Repeated random subsampling validation method was used by ten times to enhance the reliability of results37<\/a>. The average of ten results was used as the final system recognition result.<\/p>\n Fig. 3: Recognition results and analysis on standard dataset tests based on bioinspired reservoir computing paradigm.<\/b> a<\/b> Examples of digital images and skeleton frames corresponding to a home-made action \u201cHigh throw\u201d and an action \u201cBasketball shoot\u201d in UTD-MHAD dataset, where the digital images and skeleton frames of home-made action were collected by authors through a mobile phone camera and a Kinect camera, respectively. Digital images and skeleton data in UTD-MHAD dataset is reprinted with permission36<\/a>, Copyright 2015, IEEE. b<\/b> The partial time sections of reservoir states triggered by the light pulse trains and collected from the IGZO synaptic transistors. c<\/b> Specific recognition results with bioinspired reservoir computing on UTD-MHAD dataset. d<\/b> The recognition accuracies of UTD-MHAD dataset standard test with different numbers (2, 3, and 4) of Gaussian receptive fields. The error bars represent the standard deviation of five independent tests. e<\/b> Recognition accuracies achieved on different validation datasets with bioinspired reservoir computing.<\/p>\n Figure\u00a03c<\/a> shows the recognition results with respect to UTD-MHAD dataset. The recognition rate based on all action classes reaches 93.58%. As can be seen, most actions can be well recognized and only a few actions like \u201cWave\u201d are with relatively low recognition accuracy. Among all action classes, the system achieves an excellent recognition rate of 100% on 14 classes, and high accuracy (\u2265 90%) on 21 classes. This shows that our system can well distinguish multiple types of complex action processes.<\/p>\n The recognition tasks were also verified with different numbers (2, 3, and 4) of Gaussian receptive fields as shown in Fig.\u00a03d<\/a>. The Gaussian centers were also sequentially set according to Eq. (6<\/a>). As shown in Fig.\u00a03d<\/a>, high recognition accuracy (>90%) on UTD-MHAD dataset is obtained by three receptive fields. This is consistent with the results of MC metrics in Fig.\u00a02f<\/a>. We further investigated the effect of standard deviation of Gaussian receptive fields on the recognition results. Our results indicate that \u03c3 can have a slight effect on recognition accuracy. By varying \u03c3 from 0.1 to 10, the maximum variation on recognition accuracies is only 1.11% (Supplementary Fig.\u00a021<\/a>). Computation complexity can be reduced based on such GRF encoding scheme. On the one hand, the scale of our network parameters are almost 2 orders of magnitudes smaller than previous machine learning algorithms (e.g., ResNet18) regarding UTD-MHAD dataset (detailed illustration can be found in Supplementary Fig.\u00a022<\/a>)38<\/a>,39<\/a>,40<\/a>. On the other hand, multiple training iterations are not required in our training process, and only one-step linear regression is needed, which also reduces the computation complexity. Our results indicate that GRF-based preprocessing is vital important for implementing the human action recognition by reservoir computing, and the very limited number of GRF neurons can have a significant improvement in recognition accuracy.<\/p>\n We further verified this bioinspired reservoir computing paradigm on human action recognition tasks by three datasets of MSR Action3D, Florence 3D, and MSR Action Pairs. The MSR Action3D dataset contains 20 classes of skeleton-based action frames, and 20 skeleton joints\u2019 coordinates are recorded per frame41<\/a>. The dataset was used for verification, where 90% samples were chosen randomly for training and the rest for testing. A high recognition rate of 90.50% is obtained (Supplementary Fig.\u00a023<\/a>). Florence 3D dataset contains 9 action classes, and 15 skeleton joints\u2019 3D coordinates are recorded in each frame42<\/a>. Our system can identify 91.11% of the testing samples successfully (Supplementary Fig.\u00a024<\/a>). The actions in MSR Action pairs dataset are in paired with similar action trajectories43<\/a>. This makes it difficult to recognize actions in a pairwise relationship. As shown in Supplementary Fig.\u00a025<\/a>, 12<\/a> action classes can be well distinguished with an average recognition accuracy of 90.67%. Specific parameters of bioinspired reservoir on three dataset standard tests can be found in Supplementary Table\u00a02<\/a>. Repeated random subsampling validation method was applied by five times in the three tasks to enhance the reliability of results. In addition, such bioinspired reservoir computing paradigm can be also applied for high-precision and multi-type action recognition task based on depth images in videos (achieving an accuracy of 92.35% on 27 labels, and the calculation process can be found in Supplementary Note\u00a04<\/a>). As high recognition accuracies (>90%) can be achieved on different validation datasets as shown in Fig.\u00a03e<\/a> and Supplementary Fig.\u00a026<\/a>, our system exhibits remarkable versatility and fault tolerance for human action recognition.<\/p>\n Recognition of falling behaviors based on analog system<\/p>\n The conceptual diagram of our Alpho-RC system for real-world human action processing tasks is shown in Fig.\u00a04a<\/a>. Skeleton frames of daily activities collected by the Microsoft Kinect camera are the original input, and the Gaussian receptive field based encoding mechanism is implemented. During the training process, currents obtained by population transistors in response to encoding pulse trains are used as reservoir states for training output weights. During the testing process, resulting output weights are mapped by 1T1R array, and output currents of testing action sample are obtained. The video processing tasks are mostly involved in human action recognition and prediction. Human action recognition tasks usually focus on completed actions, and achieve classification purposes by learning the entire processes of actions. However, in actual scenarios, it is often necessary to achieve early classification and prediction before action ends, such as predicting and alerting before falling behaviors are completed44<\/a>. Purpose of the prediction task is to output the corresponding action label by using only the early frame sequences for inferencing.<\/p>\n Fig. 4: Recognition of falling behaviors based on Alpho-RC system.<\/b> a<\/b> The computational architecture schematic diagram of Alpho-RC system for human action recognition and prediction. b<\/b> Examples of five actions including two normal actions (squat and stretch) and three falling actions (fall down, fall to the left and fall to the right) recorded by a mobile phone camera. c<\/b> Examples of action skeleton frames corresponding to five actions. d<\/b> Device conductance from 200\u22121300\u2009\u00b5S mapped from the numerical weight values. e<\/b> The write error tolerance of \u00b11% during mapping program. f<\/b> Recognition accuracy of 96.67% obtained by memristor-based output layer. g<\/b> Comparison of energy consumption per action among Alpho-RC system and algorithms based on various advanced processors.<\/p>\n Falling behaviors as common events in real life often involves potential health risks. For example, elderly people living alone and children are more likely to suffer physical injuries if they fall, thus affecting their health. Effectively identify falling behaviors from normal behaviors is crucial to the safety of both young and old45<\/a>,46<\/a>. A home-made 3D skeleton-based falling dataset was used for verifying the efficacy of such system for dealing with real-world tasks. Such dataset including two normal actions (squat and stretch) and three falling actions (fall down, fall to the left and fall to the right) was collected and built by both digital and Kinect camera. Figure\u00a04b<\/a> shows examples of the five actions recorded by a mobile phone camera. Figure\u00a04c<\/a> shows examples of action skeleton frames recorded by a Kinect camera correspondingly. Specific demonstration of skeleton frames can be found in Supplementary Video\u00a0II<\/a>. The collection details can be found in the \u201cMethods\u201d. Here, the recognition on such home-made falling dataset was implemented by our Alpho-RC system. The 90% of action samples in the dataset were randomly selected as the training set, and the output weights were obtained by noise-aware linear regression training method (M\u2009=\u20096, L\u2009=\u20095). The detailed training process can be found in Supplementary Note.\u00a05<\/a>. The offline-trained weights were then programmed into the memristor cells in the 1T1R array. The numerical weight values were firstly mapped to device conductance from 200-1300 \u00b5S (Fig.\u00a04d<\/a>) and then programmed into the array using a write-and-verify programming scheme. Detailed reliability data of the hardware setup can be found in Supplementary Figs.\u00a027<\/a>, 28<\/a>. Good programming accuracies were achieved with write error tolerance of \u00b11% (Fig.\u00a04e<\/a>). The memristor-based output layer successfully classified five actions with recognition accuracy of 96.67% (as shown in Fig.\u00a04f<\/a>), which is close to software simulations of 98.33% (as shown in Supplementary Fig.\u00a029<\/a>). The successful demonstration of human action recognition in the hardware based bioinspired in-materia reservoir computing paves the way for energy-efficient edge computing applications. Our Alpho-RC system was also applied for human action prediction. Only the observed frames were fed into the photoelectronic reservoir for predication. Here, the ratio between the number of observed frames and the frames of the completed action is defined as observation ratio. As shown in Supplementary Fig.\u00a030<\/a>, the prediction accuracies are plotted as function of the observation ratios with respect to home-made falling dataset. The prediction accuracies are larger than 80% with the observation ratio higher than 50% and larger than 90% with the observation ratio higher than 70%. Such results indicate our system can achieve excellent prediction results, verifying the capability of both human action recognition and prediction.<\/p>\n Finally, the energy efficiency of such system is evaluated by calculating the energy consumption corresponding to processing one completed action. As mentioned before, no complex filtering process is required for the feature extraction of every action. The most energy consumption before the training of weights connected to the output layer should be attributed to encoding process and the current flow through transistors. The energy consumption for processing an action by our system is only ~45.78\u2009\u03bcJ (detailed estimation can be found in experimental section). However, feature extraction is required in most previous software-based methods, which dramatically increases the floating-point operations and the energy consumption in turn. The comparison of energy consumption is summarized in Fig.\u00a04g<\/a>. Our Alpho-RC system is at least two orders of magnitude lower than that of CMOS-based processors including CPU, FPGA, and GPU47<\/a>,48<\/a>,49<\/a>. We also compared the reported physical reservoir computing architectures for human action recognition (Supplementary Table\u00a03<\/a>), achieving largest type number of human action classification13<\/a>,50<\/a>,51<\/a>,52<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"An analog photoelectronic reservoir computing system Figure\u00a01a shows the dynamic vision processing in the human visual system. Visual…\n","protected":false},"author":2,"featured_media":56149,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3164],"tags":[3284,29693,3965,29694,3966,70,53,16,15],"class_list":{"0":"post-56148","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-computing","8":"tag-computing","9":"tag-electronic-devices","10":"tag-humanities-and-social-sciences","11":"tag-information-storage","12":"tag-multidisciplinary","13":"tag-science","14":"tag-technology","15":"tag-uk","16":"tag-united-kingdom"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@uk\/114412876368970025","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/56148","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/comments?post=56148"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/56148\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media\/56149"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media?parent=56148"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/categories?post=56148"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/tags?post=56148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"figure](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2025\/04\/41467_2025_56899_Fig1_HTML.png\")
<\/a><\/p>\n![\"figure](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2025\/04\/41467_2025_56899_Fig2_HTML.png\")
<\/a><\/p>\n![\"figure](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2025\/04\/41467_2025_56899_Fig3_HTML.png\")
<\/a><\/p>\n![\"figure](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2025\/04\/41467_2025_56899_Fig4_HTML.png\")
<\/a><\/p>\n