Event-based 3D scanning<\/p>\n
Our system uses neuromorphic event cameras40<\/a>,41<\/a>,42<\/a>,43<\/a>, biologically inspired vision sensors that output a stream of events rather than complete image frames. A pixel is triggered, and an event is read out only if the log of its intensity I changes by more than a set threshold \u03f5 (controlled by the bias parameters of the camera) in an interval \u0394t (dictated by a number of factors, including bias parameters and rate of intensity change at that pixel). A collection of these events forms the event stream. Each event eijt in the stream is asynchronously transmitted as a tuple of pixel coordinates (i,\u00a0j), the associated timestamp t, and the polarity p (positive or negative) of the intensity change, <\/p>\n $${e}_{ijt}=(i,j,t,p),s.t.| \\log (I(i,j,t))-\\log (I(i,j,t-\\Delta t))| > \\epsilon .$$<\/p>\n \n (1)\n <\/p>\n The sparse read-out scheme of event cameras allows imaging fast-changing scenes with latencies down to the order of 10\u2009\u03bcs, depending on event sparsity40<\/a>,41<\/a>,42<\/a>,43<\/a>. The differential nature of event cameras also drastically increases their dynamic range and robustness to ambient light compared to frame-based cameras, which require adjustment of exposures for different lighting conditions and surface types.<\/p>\n As discussed, 3D imaging with event cameras has so far been mainly limited to event-based triangulation for diffuse surfaces. The basic idea is that multi-shot triangulation principles with sparse patterns (such as point raster scanning or line sweeping using lasers) can be significantly sped up by using a low-latency event camera44<\/a>,45<\/a>,46<\/a>,47<\/a>,48<\/a>. Figure\u00a02<\/a>b depicts the basic principle. A correspondence between the projector pattern and the camera pixels is established using the timestamps of the signals detected at the respective event camera pixels. This correspondence is then used to obtain a depth value for each camera pixel via triangulation.<\/p>\n Our method uses a combination of two perpendicular line lasers to scan the scene by alternating between horizontal and vertical sweeps. We use these measurements to computationally synthesize the equivalent to a point scanning system, where each point position is obtained from the intersection of the scanned vertical and horizontal lines (see Fig.\u00a03<\/a>a). This procedure provides a significant speed advantage over single-point raster scanning. Assuming a square (two-dimensional) scan area whose sweep across one side-length takes time T, raster scanning requires T2 total time, whereas our dual orthogonal sweeps take only 2T (see Supplementary\u00a0S3.1<\/a> for more details).<\/p>\n Fig. 3: Dual-scan line projector and experimental setup.<\/b> a<\/b> We sweep a line laser horizontally (i) and then vertically (ii). To obtain the events for a particular projector pixel, we take an intersection of these vertical and horizontal line events (iii). This allows us to have a pixel-to-pixel correspondence for specular surface reconstruction via deflectometry. b<\/b> Our setup consists of a Prophesee EVK4 event camera and a dual-scanning laser projector. The dual-scanning laser projector is made from a crosshair (+) laser diode (CivilLaser crosshair laser diode, 80\u2009mW, 532\u2009nm), a galvo scanning system (Thorlabs GVS012), and a Digital Acquisition Board (NI DAQ PCIe 6363) to control the galvo mirrors. The baseline for our triangulation system is 30\u2009cm, with a triangulation angle of 26\u00b0.<\/p>\n We note that a sweep in two directions is not necessary for triangulation on purely diffuse scenes, as a single line sweep in conjunction with epipolar geometry is sufficient to uniquely obtain camera-projector correspondences. However, there is no epipolar constraint in deflectometry, meaning that a point scanning equivalent (sweep in two directions) is essential for epipolar separation and its subsequent deflectometry evaluation.<\/p>\n Diffuse-specular separation<\/p>\n Assuming the measured scene contains only diffuse surfaces and that no interreflections occur, each point in the projection pattern maps to exactly one point in the camera image, resulting in a direct correspondence. In mixed reflectance scenes, a mix of diffuse and specular reflections as well as other phenomena such as subsurface scattering leads to signal ambiguities that need to be resolved. In particular, our approach uses triangulation to evaluate the 3D shape from diffuse reflection events (Fig.\u00a02<\/a>e) and deflectometry for their specular counterpart (Fig.\u00a02<\/a>f). This requires us to separate the events into single-bounce (direct, triangulation), two-bounce (indirect, deflectometry), and other indirect multibounce reflections.<\/p>\n Our separation algorithm exploits epipolar geometry between the laser projector and the event camera to distinguish direct signal components from indirect illumination. Events located along the epipolar line are considered direct reflection components and are evaluated via triangulation. Events located outside the epipolar line are considered indirect reflection components, which contain the two-bounce events that are evaluated via deflectometry. We separate these two-bounce events from the other indirect components by applying additional scene constraints, e.g., by checking if a direct reflection component is present at the same time a two-bounce component event candidate is detected. A detailed explanation of the applied classification rules is outlined in the \u201cMethods\u201d section, and the Supplementary Section\u00a0S2.1<\/a>.<\/p>\n Some partially specular objects like a plastic cup show both diffuse and specular reflections. For a particular pixel covering this object, it might get a diffuse reflection for some timestamp and specular for another. In such cases, we classify that pixel as diffuse and reject its specular component. We then use triangulation for its shape estimation, as we do for all the diffuse components. We can do this because of the high dynamic range of the event camera, as we still get a good signal if we only evaluate the (sometimes very dim) diffuse component. This is another benefit of using event cameras.<\/p>\n 3D scene reconstruction: everything around is a screen<\/p>\n Conventional deflectometry systems typically rely on an external screen as an illumination source, which introduces two major limitations. First, the geometric relationship between the screen and the camera must be accurately pre-calibrated to determine the screen coordinates. This reduces system flexibility since the screen must remain fixed after calibration. Second, the effective measurement area of specular surfaces (i.e., its coverage) is fundamentally constrained by the screen\u2019s size, shape, and position relative to the camera and object. To reconstruct 3D scenes with our approach, we first evaluate the direct reflection events via triangulation to obtain the 3D coordinates of the diffuse surfaces in the scene. These reconstructed diffuse surfaces are then repurposed as virtual screens as follows: After the diffuse components have been evaluated via triangulation, the 3D position (x, y, z) of the signal (the point generated by the dual-line scan, see Fig.\u00a03<\/a>) on the respective surface is known for every timestamp. This position is also observed over the specular surfaces in the scene and serves now as a virtual screen pixel for the deflectometry measurement. Together with the respective two-bounce event detected at this timestamp, the virtual screen pixel constitutes the correspondence pair to be evaluated with our deflectometry algorithm. This procedure completely eliminates the need for an externally calibrated screen. Because any diffuse surface in the environment can serve as a virtual screen, everything around becomes a screen, which has the potential to significantly increase the effective deflectometry measurement area coverage compared to conventional deflectometry systems with fixed screens of finite size.<\/p>\n To estimate the surface normals of specular surfaces in our deflectometry evaluation (Fig.\u00a02<\/a>f), we formulate a joint optimization problem that infers depth from the measured correspondences between virtual screen pixels and two-bounce events. In this iterative process, the algorithm minimizes the cosine similarity loss between (i) normals obtained from the correspondence map (formed by the bisector between the rays from virtual screen pixel to surface point, and surface point to camera pixel), and (ii) normals computed from the gradient of the reconstructed surface. The optimized depth from this procedure yields the final specular shape. Details are described in the \u201cMethods\u201d section and Algorithm 2. In a last step, we combine the diffuse and specular surface reconstructions in one coordinate system to obtain the full 3D shape of the mixed reflectance scene. In the following, we present the performance of our framework in real-world experiments under various conditions.<\/p>\n ExperimentsImaging mixed reflectance scenes<\/p>\n We calibrate a measurement volume of ~25\u2009cm\u00a0\u00d7\u00a045\u2009cm\u00a0\u00d7\u00a060\u2009cm and place diffuse, specular, and partially specular objects within this space. Our setup (see Fig.\u00a03<\/a>b) consists of a Prophesee EVK4 event camera (see Supplementary Section\u00a0S3.3<\/a> for bias parameters adjustment) with a dual-scanning laser projector made from CivilLaser 80 mW, 532\u2009nm crosshair (+) laser, Thorlabs GVS012 galvo scanner, and connected with a NI (National Instruments) DAQ (Digital Acquisition Board) PCIe 6363. Figures\u00a04<\/a> and 5<\/a> show our reconstructions for a variety of mixed reflectance scenes.<\/p>\n Fig. 4: Reconstructions of mixed reflectance scenes and comparison with state-of-the-art and commercial 3D-sensing methods.<\/b> Our reconstructions (second column) are compared with ESL45<\/a> (third column), Ensemble Codes33<\/a> (fourth column), Intel RealSense (fifth column), and Microsoft Kinect v2 ToF (sixth column). The scanning times \u03c4 are indicated at the top of each column. a<\/b> Diffuse plane with a specular ball bearing, causing multiple interreflections. After the diffuse-specular separation, we use triangulation to obtain the direct components (green, including the small portion of the ball in front) and deflectometry for the indirect components (the dark\/red region of the ball). b<\/b> Specular balloon dog with diffuse plane. c<\/b> Specular bird with diffuse plane. d<\/b> Shiny license plate with a diffuse plane. The insets in (a<\/b>\u2013c<\/b>) show the reconstructed specular shapes from different viewpoints to illustrate the reconstruction quality. (D direct component using triangulation, S specular component using deflectometry). Summary of observations: Our method successfully reconstructs both diffuse and specular surfaces with high quality. In contrast, the performance of other methods is limited in the presence of specular reflections.<\/p>\n Fig. 5: Additional reconstructions of mixed reflectance scenes and comparison with state-of-the-art and commercial 3D-sensing methods for scenes from Fig. 1a\u2013d<\/a>.<\/b> Our reconstructions (second column) are compared with ESL45<\/a> (third column), Ensemble Codes33<\/a> (fourth column), Intel RealSense (fifth column), and Microsoft Kinect v2 ToF (sixth column). For the reference methods, the scanning times \u03c4 are fixed and indicated at the top of each column, whereas our scanning time varies depending on the demonstrated measurement modality. a<\/b> Mixed reflectance scene with multiple interreflections. b<\/b> Mixed reflectance scene with moving objects (shiny balloon and paper towel) and static objects (bust, Stanford bunny53<\/a>, mirror, and background). c<\/b> Scene with purely diffuse surfaces, at a reduced scanning time of \u03c4\u00a0=\u00a030\u2009ms. d<\/b> Scene with purely diffuse surfaces captured at a fast scanning time of 4\u2009ms. Summary of observations: ESL can reconstruct diffuse scenes but shows limited performance in the presence of specular surfaces, and the overall reconstruction quality is lower than ours. Ensemble Codes achieve good results on diffuse regions but provide little information on specular parts. RealSense and Kinect v2 produce low-quality depth maps on both diffuse and shiny regions and often yield inaccurate or missing data on specular surfaces. Our method reconstructs both diffuse and specular surfaces with high quality. Even with an acquisition time of 4\u2009ms, our method produces reconstructions of higher quality than RealSense and Kinect v2, both captured at \u03c4\u00a0=\u00a033\u2009ms.<\/p>\n We compare our method against four baselines consisting of commercially available products as well as our own implementations of state-of-the-art methods: (1) Event-based Structured Light (ESL)45<\/a>\u2014a recent method for event-based triangulation, (2) Ensemble Codes33<\/a>\u2014a structured triangulation framework designed to show improved robustness to interreflections from diffuse surfaces, (3) Intel RealSense D435i\u212250<\/a>\u2014a commercially available triangulation-based structured light 3D scanner, and (4) Microsoft Kinect\u2122 v251<\/a> a commercially available time-of-flight (ToF) 3D sensor.<\/p>\n For our ESL implementation, we use the same event camera (Prophesee EVK4) along with an Anybeam MEMS laser scanning pico-projector, scanning at 60\u2009Hz, resulting in a capture time of \u03c4\u00a0=\u00a016\u2009ms per 3D view. Ensemble Codes requires 50 projector-camera frames per 3D reconstruction. Using our hardware (FLIR BFS-U3-19S4C camera and Viewsonic X11-4K projector), an exposure time of 70\u2009ms per frame yields a total capture time of \u03c4\u2009=\u20093.5\u2009s per 3D view. RealSense and Kinect operate at 30\u2009Hz (\u03c4\u2009=\u200933\u2009ms per view). For mixed reflectance scenes, our method achieves reliable low noise reconstructions using a 30\u2009ms scan per line direction with a 5\u2009ms recovery interval between them, resulting in \u03c4\u00a0= (30\u2009ms\u2009+\u20095\u2009ms) \u00a0\u00d7\u00a02\u2009=\u200970\u2009ms per 3D view. This results in a 3D framerate of ~14\u2009Hz for mixed reflectance scenes (see additional sensor details and comparisons in Tables S1<\/a> and S2<\/a>).<\/p>\n In Fig.\u00a04<\/a>a, a specular sphere generates strong interreflections when illuminated with the laser projector. Our method leverages both direct diffuse and direct specular reflections (from the sphere center) for triangulation (marked \u201cD\u201d in Fig.\u00a04<\/a>) and secondary reflections for deflectometry (marked \u201cS\u201d in Fig.\u00a04<\/a>), which results in the 3D recovery of a large portion of the visible surface, illustrating the increased coverage enabled by the shown multi-surface screen geometry. In Fig.\u00a04<\/a>b\u2013d, we illuminate only a planar diffuse screen and reconstruct specular objects solely from second-bounce reflections.<\/p>\n Additional measurement examples in Fig.\u00a05<\/a> further demonstrate the reconstruction of specular objects with complex shapes embedded in mixed reflectance scenes. Figure\u00a05<\/a>a shows a mixed reflectance scene containing a convex mirror and a diffuse Michelangelo\u2019s David bust. Despite strong interreflections, our method reconstructs both diffuse (gray) and specular (blue) surfaces with high quality. Ensemble Codes reconstructs the diffuse regions with high quality but fails to recover the specular mirror surface. ESL, RealSense, and Kinect produce incomplete or distorted depth at the mirror location and lower-quality diffuse reconstructions.<\/p>\n Figure\u00a05<\/a>b demonstrates the reconstruction of a moving mixed reflectance scene comprising a partially specular (glossy) balloon, a specular mirror, and multiple objects with diffuse surfaces. Our method performs motion-robust reconstruction at 14\u2009Hz (see Supplementary Movie\u00a01<\/a>), simultaneously recovering the shape of specular, glossy, and diffuse surfaces. Ensemble Codes reconstructs only stationary diffuse regions and misses specular surfaces and moving scene parts. ESL, RealSense, and Kinect capture motion but fail to recover accurate mirror geometry, and the latter two also exhibit lower fidelity of diffuse surface parts.<\/p>\n To the best of our knowledge, this marks the first-ever demonstration of event-based deflectometry. Instead of an actively back-illuminated display, the method works with a passive screen (a piece of cardboard). This screen does not need to be flat (can have an arbitrary shape) and does not need to be calibrated into the system, as its position and shape are evaluated on-the-fly via triangulation for every 3D frame. This has interesting consequences for challenging deflectometry measurements that require even more coverage: for stationary specular objects, a piece of cardboard can be moved around the object during a measurement sequence, effectively emulating multiple screens placed at multiple positions. Eventually, a 3D model with improved coverage can be generated by fusing the event-deflectometry measurements from each screen position. Early results of this approach are shown in Supplementary Fig.\u00a0S7<\/a>. Additional results for the measurement of mixed reflectance scenes are shown in Supplementary Fig.\u00a0S4<\/a>.<\/p>\n Imaging scenes with purely diffuse surfaces<\/p>\n A major part of event-based 3D scanning literature focuses on 3D reconstruction of scenes with purely diffuse surfaces via triangulation44<\/a>,45<\/a>,46<\/a>,47<\/a>,48<\/a>,49<\/a>. Our method also allows for measuring these scenes while providing additional degrees of freedom with respect to choosing the scanning speed.<\/p>\n As discussed, the scanning time can be further reduced when working with purely diffuse surfaces by performing a single laser line sweep rather than the dual-scanning required for deflectometry. This allows us to reduce scanning time by 2\u00d7 without any loss in reconstruction quality for diffuse scenes. A comparison of our \u03c4\u00a0=\u00a030\u2009ms diffuse-only reconstruction with other approaches is shown in Fig.\u00a05<\/a>c. Our method achieves visually higher data quality than the \u03c4\u00a0=\u00a03.5\u2009s ensemble code scan, and shows significant improvement over ESL, Kinect, and RealSense. In diffuse-only scenarios, the scanning speed can be further increased as multibounce cases do not have to be classified. In our prototype setup, we experimentally reduced the capture time to just \u03c4\u00a0=\u00a04\u2009ms while still achieving high-quality reconstructions. A comparison of our \u03c4\u00a0=\u00a04\u2009ms capture with the other approaches is shown in Fig.\u00a05<\/a>d. Although slightly lower quality than the \u03c4\u00a0=\u00a03.5\u2009s ensemble code scan, and our previous \u03c4\u00a0=\u00a070\u2009ms or \u03c4\u00a0=\u00a030\u2009ms scan, our fast \u03c4\u00a0=\u00a04 ms scan still shows significantly better data quality than RealSense or Kinect (both at \u03c4\u00a0=\u00a033\u2009ms). This shows that our method has significant potential for future low-latency 3D scanning implementations\u2014even if only standard diffuse scenes are measured. To further visualize the degradation of the reconstructed 3D point clouds with decreasing scan speeds, we show our reconstructions of diffuse surfaces at different scan times from \u03c4\u00a0=\u00a030\u2009ms to up to \u03c4\u00a0=\u00a04 ms per frame or 250\u2009Hz in Fig.\u00a06<\/a>a, b. This marks one of the fastest scans shown to date for an event-based structured light system.<\/p>\n Fig. 6: Evaluation of reconstruction quality vs. scan time.<\/b> a<\/b> 3D measurements of purely diffuse scene at \u03c4\u00a0=\u200930\u2009ms, 16\u2009 ms, 8\u2009ms, and 4\u2009ms. b<\/b> Profile plots at two different positions of the bust. The 3D reconstructions display more noise as the scan time decreases. c<\/b> Mixed reflectance scene (planar diffuse surface with partially specular lens case made of clear plastic), measured at \u03c4\u00a0=\u200970\u2009ms, 40\u2009ms, 20\u2009ms, and 12\u2009ms. Faster scan times result in a reduction in the intensity of secondary light reflections from specular objects, to the point where these changes are not detected as events anymore. Hence, the coverage and reconstruction quality decrease as scan time decreases.<\/p>\n Tradeoff between diffuse surface reflectivity and specular coverage<\/p>\n Lastly, we also demonstrate the performance of our system for a mixed reflectance scene as the scan time reduces in Fig.\u00a06<\/a>c. We use a plastic lens case that is less reflective and more transmissive than a mirror. As the scan becomes faster, the signal intensity for the diffuse bounce and hence its secondary reflection from specular parts becomes lower. The intensity of the second bounce specular reflection eventually becomes lower than the change threshold that the event camera can detect. Hence, its coverage starts vanishing for lower scanning times. Using a laser with higher output power could potentially improve the result.<\/p>\n Similar to Eq. (1<\/a>), an event corresponding to a second bounce reflection from a specular surface S, denoted by \\({e}_{ijt}^{S}\\) is given by, <\/p>\n $${e}_{ijt}^{S}=(i,j,t,p),s.t.\\Delta \\log {I}_{ijt}^{S} > \\epsilon,$$<\/p>\n \n (2)\n <\/p>\n where \\(\\Delta \\log {I}_{ijt}^{S}=| \\log ({I}^{S}(i,j,t))-\\log ({I}^{S}(i,j,t-\\Delta t))| \\) is governed by <\/p>\n $$\\Delta \\log {I}_{ijt}^{S}\\propto \\tau \\cdot {P}_{L}\\cdot {r}_{D}\\cdot {r}_{S}.$$<\/p>\n \n (3)\n <\/p>\n Here, \u03c4 is the scanning time, PL is the laser power, and rD and rS denote the fraction of light reflected from the diffuse screen and the specular surface, respectively. rD and rS are functions of the diffuse and specular surface\u2019s bidirectional reflectance distribution functions (BRDFs).<\/p>\n Reduced scanning time \u03c4 will lead to a reduced probability of secondary event detection, as can be seen in Fig.\u00a06<\/a>c. A darker reflecting diffuse surface (small rD) would also reflect a lesser fraction of light onto the secondary specular surface, leading to the same effect. This is also true for a smaller reflectivity of the secondary specular surface rS. If it\u2019s highly transmissive (like the plastic lens case) or highly absorptive, then the probability of event detection falls proportionally.<\/p>\n Imaging dynamic scenes<\/p>\n One of the advantages of our system is fast high-quality capture which allows for motion-robust reconstructions of moving mixed reflectance scenes (mixed reflectance 3D videos). Representative frames from five different scenes in these videos are shown in Fig.\u00a07<\/a>, and the full videos are available in the Supplementary Movie\u00a01<\/a>. At a \u03c4\u00a0=\u00a070\u2009ms scan time (14\u2009Hz), we reconstruct mixed reflectance scenes containing moving specular, partially specular, and diffuse object surfaces, including a glossy balloon and a convex mirror in Fig.\u00a07<\/a>a, and plastic lens case that is closed by a hand in Fig.\u00a07<\/a>e. When scanning only diffuse or glossy surfaces Fig.\u00a07<\/a>b\u2013d, our system does not require deflectometry evaluation. This lets us exploit epipolar constraints for the triangulation measurements, which requires a projector line sweep in only one direction and increases the 3D acquisition frame rate to 29\u2009Hz (\u03c4\u00a0=\u00a035\u2009ms per 3D frame). Figure\u00a07<\/a>b shows the motion-robust reconstruction of a bouncing table tennis ball. Rapid ambient illumination changes, induced by a moving bright light source in Fig.\u00a07<\/a>c, do not affect reconstruction fidelity, highlighting the high dynamic range and robustness to ambient light across both dark (the side of the bulb) and brightly lit regions (David bust). Figure\u00a07<\/a>d shows a successful reconstruction of a glossy treasure chest with partially specular surfaces (such as the metal strip and handle).<\/p>\n Fig. 7: Motion-robust 3D measurements of moving scenes.<\/b> (see Supplementary Movie\u00a01<\/a>). a<\/b> 14 Hz reconstruction of a mixed reflectance scene containing a specular mirror, a shiny balloon, and other diffuse objects (David bust, Stanford bunny53<\/a>, paper towel). We show intermediate frames from the video where the shiny balloon is rotated, and the paper towel is rapidly lifted. b<\/b> 29\u2009Hz reconstruction of a bouncing table tennis ball. c<\/b> 29\u2009Hz reconstruction of a scene with a moving light bulb causing strong changing background illumination. Despite varying ambient lighting in the scene, we reconstruct the shapes of both the bright David bust and the dark side of the light bulb. d<\/b> 29\u2009Hz reconstruction of a closing treasure chest that has diffuse as well as partially specular surface components (the metallic lock and the strip); our method reconstructs both surface types. e<\/b> 14\u2009Hz reconstruction of closing a partially specular clear plastic lens box by hand, where its bottom half is fixed. In the intermediate frames, we can see the transition from seeing the specular top half to later seeing the diffuse fingers as the box closes. (D direct component using triangulation, S specular component using deflectometry, Red arrows indicate motion).<\/p>\n Quantitative and qualitative evaluation<\/p>\n We evaluate the accuracy and precision of our 3D reconstructions using objects with known geometry. For event-based triangulation on diffuse surfaces, we measure a precision-manufactured diffuse planar board and a diffuse sphere (radius 25.4\u2009mm). The objects are placed at a working distance of \u00a0~60\u2009cm and measured with a \u03c4\u00a0=\u00a070\u2009ms scan time via our dual-laser scan. Reconstructed point clouds are compared to the corresponding ground-truth shapes.<\/p>\n Diffuse reconstruction (event-triangulation)<\/p>\n The accuracy of our method is evaluated by calculating the root-mean-square error (RMSE) of our 3D reconstruction with respect to the ground-truth surface. The planar surface reconstruction yields an RMSE of 210\u2009\u03bcm, and the diffuse sphere yields an RMSE of 310\u2009\u03bcm. To evaluate the precision of our method, we isolate the statistical noise on our data by subtracting a low-frequency best-fit surface and compute the standard deviation of the residual 3D noise. The measured precision is 120\u2009\u03bcm for the planar surface, and 60\u2009\u03bcm for the sphere.<\/p>\n Specular reconstruction (deflectometry)<\/p>\n We evaluate our deflectometry reconstruction in a similar fashion by measuring a specular sphere with known radius (25.4\u2009mm), using the diffuse plane as the virtual screen (setup similar to Fig.\u00a04<\/a>a). After rejecting spurious outliers outside a 6\u03c3 interval (4.8% of data), the accuracy is evaluated with an RMSE of 450\u2009\u03bcm, with a precision of 140\u2009\u03bcm.<\/p>\n A deflectometry measurement in our framework includes a preceding triangulation reconstruction of the diffuse virtual screen, whose position uncertainty propagates into the specular shape estimate. Our error propagation analysis indicates that the resulting lateral screen-position uncertainty from our triangulation measurement is ~10\u00a0\u00d7 larger than the lateral back-projection screen error in a well-calibrated high-precision deflectometry setup17<\/a>, explaining the ~100\u2009\u03bcm-scale accuracy of our deflectometry measurements compared to the single-digit \u03bcm accuracies (or sometimes even sub-\u03bcm accuracies) of high-performance deflectometry setups.<\/p>\n Relative distance benchmark (comparison to prior event-based structured light)<\/p>\n To enable comparison to prior event-based structured light evaluations on diffuse surfaces44<\/a>,45<\/a>, we perform a relative translation experiment with a planar target. Prior work often reports depth error by comparing the estimated absolute plane distance dest to a ground-truth distance dgt. However, accurately measuring absolute distance at the tens-of-microns level that would be required for our purpose is extremely challenging and unreliable, motivating a relative translation distance benchmark. The planar target is mounted on a translation stage (Zaber XLHM-100A, 100\u2009mm range, 0.12\u2009\u03bcm resolution, see Fig.\u00a08<\/a>b) at ~500\u2009mm standoff distance. The stage position at 50\u2009mm is used as reference (relative displacement 0\u2009mm), and the plane is translated in the \u00a0\u00b1\u00a050\u2009mm range in 10\u2009mm steps. We first fit a plane to the reference position measurement and computationally translate this virtual plane by the known displacement \\(\\Delta {d}_{gt}^{i}\\) to obtain the expected plane equation and compute the expected depth \\({d}_{\\exp }^{i}(p)\\) at each pixel p in the region of interest (ROI). Next, we measure the actual reconstructed depth map \\({d}_{est}^{i}(p)\\) for plane i and calculate the root-mean-square error over all P pixels in the ROI.<\/p>\n $${{\\rm{RMSE}}}^{i}=\\sqrt{\\frac{1}{P}{\\sum }_{p=1}^{P}{({d}_{{\\rm{est}}}^{i}(p)-{d}_{\\exp }^{i}(p))}^{2}}.$$<\/p>\n \n (4)\n <\/p>\n Fig. 8: Quantitative and qualitative comparison with state-of-the-art event-based structured light systems.<\/b> Our method shows improved performance over the tested ESL implementation. a<\/b> Triangulation accuracy by measuring relative distances between planes. A white plane board is mounted on a motorized translation stage (left) and moved in 10 mm steps over a 100\u2009mm range. The 50\u2009mm position generates the reference plane equation, from which other reference planes are generated (by translating the reference plane equation). The plot (right) shows errors at each position, computed as the difference (root mean square error (RMSE)) between the reference and measured values. For a fair comparison, measurements are taken at 16\u2009ms, the same speed as ESL, demonstrating sub-mm (\u00a0<\u00a00.6\u2009mm) accuracy with our method. The vertical bars show each measurement\u2019s precision value (standard deviation). b<\/b> Point clouds of 3D reconstructions obtained from our method and ESL are shown. Both systems have identical triangulation angles. The point clouds for our approach are higher quality and of higher fidelity than those obtained from ESL, for all the speeds of our system\u2014 from 70\u2009ms mixed reflectance scans to 16\u2009ms diffuse scans. Since ESL cannot recover specular shapes, its reconstruction in the mixed reflectance scene is affected by the specular reflections (see Supplementary Section\u00a0S4.2<\/a> for more measurements).<\/p>\n For a fair comparison, we perform our method at a 16\u2009ms diffuse scan speed to match the ESL acquisition speed, leading to a slight increase in error from the measurement of the diffuse planar surface performed at 70 ms in the previous section. Figure\u00a08<\/a>a shows the measured \\(\\Delta {d}_{est}^{i,ref}\\) as i varies from \u221250 to +50\u2009mm. Our RMSE remains below 600\u2009\u03bcm across the entire translation range. Under identical conditions the reproduced ESL setup45<\/a> (see Supplementary Section\u00a0S4.2<\/a>), produces RMSE depth errors \u2272 6\u2009mm.<\/p>\n Quantitative evaluation summary<\/p>\n Across diffuse and specular reconstructions, our depth error remains \u00a0<\u00a00.6\u2009mm in these experiments, with an order-of-magnitude lower error than the ESL baseline (see Fig.\u00a08<\/a>a), while enabling mixed reflectance 3D imaging via a self-calibrated, flexible screen approach. Table S1<\/a> summarizes our performance relative to prior methods.<\/p>\n Qualitative comparison with state-of-the-art event-based structured light scanning<\/p>\n Event-based triangulation has demonstrated high-speed reconstruction for diffuse surfaces44<\/a>,45<\/a>,46<\/a>,47<\/a>,48<\/a>. As discussed, we reproduced the ESL pipeline45<\/a> using our event camera (Prophesee EVK4) and a pico-projector (Anybeam MEMS laser projector) under a triangulation angle, optics, and field of view that is matched to ours, to benchmark diffuse reconstruction performance under identical geometric conditions.<\/p>\n Representative reconstructions of our ESL setup are shown in Fig.\u00a05<\/a> (Column 3), Fig.\u00a04<\/a> (Column 3), and comparisons of raw point clouds in Fig.\u00a08<\/a>b (see Supplementary Fig.\u00a0S2<\/a> for more point clouds). While both approaches rely on triangulation, our method yields higher diffuse reconstruction fidelity and improved quantitative accuracy (see \u201cQuantitative evaluation\u201d section above). In mixed reflectance scenes, ESL does not explicitly separate interreflections or specular components and therefore fails to reconstruct specular surfaces.<\/p>\n The observed accuracy improvement arises from two primary factors: (1) improved correspondence localization, and (2) more accurate camera-projector calibration. Our continuous line scanning, combined with clustering-based localization, enhances sub-pixel correspondence precision compared to discrete scanning in ESL, whereas dual-scanning provides cleaner correspondences for better inverse calibration, as opposed to gray-code-based calibration in ESL (see Supplementary\u00a0S3.2<\/a> for details). Experimentally, our stereo calibration achieves a reprojection error of 0.4 pixels, compared to 3.6 pixels reported in ref. 45<\/a>, leading to significantly reduced depth uncertainty. Further implementation details are provided in Supplementary Section\u00a0S4.2<\/a>.<\/p>\n Qualitative comparison with conventional deflectometry<\/p>\n To benchmark our specular shape reconstruction against state-of-the-art deflectometry, we measured each test object using a display-based conventional phase-measuring deflectometry (PMD) setup (Fig.\u00a09<\/a>a\u2013d). We then applied our method utilizing an uncalibrated, movable virtual screen (Fig.\u00a09<\/a>e) to the same objects. Reconstructions for a convex mirror, a balloon dog, and a bird are presented in Fig.\u00a09<\/a>d,e. Qualitatively, our uncalibrated virtual screen approach (Fig.\u00a09<\/a>e) yields results comparable to those obtained with the calibrated PMD system (Fig.\u00a09<\/a>d). Furthermore, for both the balloon dog and the bird, our method provides increased surface coverage by allowing dynamic adjustment of the screen\u2019s shape and position without necessitating system recalibration. In contrast, conventional deflectometry requires full recalibration of the screen-camera geometry whenever its relative pose is modified.<\/p>\n Fig. 9: Comparison with conventional deflectometry.<\/b> a<\/b> A traditional phase-measuring deflectometry (PMD) setup built for qualitative evaluation. b<\/b> Recorded fringe patterns for the specular bird. c<\/b> Recovered phase map. d<\/b> Specular shapes reconstructed using conventional deflectometry. e<\/b> Specular shapes reconstructed using our method. The results demonstrate that our method achieves comparable performance in shape reconstruction and superior or comparable performance in surface coverage.<\/p>\n","protected":false},"excerpt":{"rendered":"Principles In our proposed approach, we tightly integrate event-based triangulation and deflectometry to simultaneously scan all surface types…\n","protected":false},"author":2,"featured_media":975516,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_share_on_mastodon":"0"},"categories":[3845],"tags":[7461,12788,3965,3966,74,70,16,15],"class_list":["post-975515","post","type-post","status-publish","format-standard","has-post-thumbnail","category-physics","tag-applied-optics","tag-electrical-and-electronic-engineering","tag-humanities-and-social-sciences","tag-multidisciplinary","tag-physics","tag-science","tag-uk","tag-united-kingdom"],"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@uk\/116612627061497871","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/975515","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/comments?post=975515"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/975515\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media\/975516"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media?parent=975515"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/categories?post=975515"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/tags?post=975515"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig3_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig4_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig5_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig6_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig7_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig8_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n![\"Fig.](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/41467_2026_72254_Fig9_HTML.png\")
The alternative text for this image may have been generated using AI.<\/p>\n