Multi-die assemblies and advanced packaging have multiplied the number of decisions that need to be made around I\/Os and interconnects, and that\u2019s only exacerbated by complex and rapidly changing markets.<\/p>\n
\u201cWith new packaging technologies, a lot of system-level analysis and budget-type analysis are important,\u201d said Hee Soo Lee, high-speed digital design segment lead at Keysight EDA<\/a>. \u201cAlso, at an engineer\u2019s level, being able to make really clean channels at the physical layer and ensuring those I\/Os get data out in a cleaner way is very important. That is where a lot of learning and adapting is happening, using new technologies, EDA solutions, and tools. The learning curve is demanding, but that\u2019s the core factor for success \u2014 getting into a new market, and getting market share out of these harsh market conditions.\u201d<\/p>\n Balancing tradeoffs This isn\u2019t a one-size-fits-all solution, however. Data movement can change over time, and it can vary by workload.<\/p>\n \u201cThere are tools that are helping, but the decisions are not at that micro level,\u201d Khan said. \u201cAgentic AI and other AI capabilities are getting added to pretty much all the tools in the design flow. In some places, they\u2019re more mature than others. The complex, physics-based AI is not all there yet. Some of these are very hard physics problems to solve in terms of system implementations, including thermal. As of today, there are some uses, but not to the degree that it can significantly accelerate things.\u201d<\/p>\n Others agree. \u201cThe nastiest I\/O design problems today show up where physics\u00a0and integration economics\u00a0collide,\u201d said Andy Nightingale, vice president of product management and marketing at Arteris<\/a>. \u201cThis includes chiplets, or multi-die in 2.5D\/3D packages, plus leading-edge compute tiles that push power density and clocking. Advanced packaging shortens interconnect distances, but it also multiplies interfaces, clock domains, power islands, and \u2018unknown unknowns\u2019 in signal\/power integrity, thermal gradients, and test\/bring-up.\u201d<\/p>\n Heterogeneous integration only compounds those issues. \u201cChiplets and 3D multi-die packaging present the hardest I\/O challenges due to heterogeneous interfaces, signal integrity constraints, and rapidly scaling bandwidth requirements,\u201d said William Wang, CEO of ChipAgents<\/a>. \u201cEngineers must understand signal integrity, power delivery, retry mechanisms, protocol stacking, and thermal-bandwidth tradeoffs as AI massively increases data movement pressure across dies.\u201d<\/p>\n Solving one problem at a time is possible, but they all need to be solved at once. \u201cThe coupling of multiple elements is the hardest challenge,\u201d said Ashish Darbari, CEO of Axiomise<\/a>. \u201cAny individual solver \u2014 thermal, mechanical, electrical \u2014 has gotten genuinely good. The problem is that the coupling is bidirectional and spans time and length scales that are orders of magnitude apart, and the tools involved don\u2019t naturally talk to each other. Signal integrity in multi-die systems has the same character. UCIe and BoW links running at 32, 48, or 64 giga-transfers per second are going into bumps whose impedance shifts with temperature and mechanical strain. Heterogeneous integration makes the bookkeeping painful \u2014 a compute die on N2, I\/O on N5, SRAM on N3, analog on N16 \u2014 different PDKs, different reliability models, different thermal coefficients. Making them produce a coherent picture under one workload is where most multi-die programs quietly lose months.\u201d<\/p>\n I\/O and interconnect designers need to make a series of choices, each impacting the next. \u201cChiplets and 3D do not just add another integration option. They multiply the number of boundaries you have to manage,\u201d said Lou Ternullo, senior director of product management for silicon IP at Rambus<\/a>. \u201cNow you are choosing where protocols terminate, where coherency lives, and what traffic stays on-package versus what has to survive board-level channels. You also inherit new physical realities, like thermal gradients, power integrity, and tighter signal budgets that can change what looks \u2018best\u2019 on a block diagram. The result is that interconnect is no longer a single choice. It is a hierarchy of choices across package, board, and rack.\u201d<\/p>\n In data center design, decisions are made in layers. \u201cYou start out knowing how much energy is delivered to a data center, to a rack, and you\u2019ve got power budgets and cooling budgets and so on that you\u2019re going to operate in,\u201d Khan said. \u201cThen there is a budget that gets handed down to a system maker. Not all the components are coming from the same company, either. The budgets are apportioned, then the person building a system may acquire multiple pieces of the equipment from various vendors. How the decisions then get made at each level is going to be a little bit different based on the technical budget and PPA, in addition to cost.\u201d<\/p>\n Distance between devices is a central consideration. \u201cThe main challenge is routing,\u201d said Satish Radhakrishnan, head of GTM for semiconductor and electronics at Vinci<\/a>. \u201cInterconnect protocols span very different physical distances, from millimeters inside a package to meters within a rack. As more devices are connected and brought closer together, routing becomes much harder. Designers must manage congestion, signal integrity, power delivery, thermal impact, and the physical limits of the package, board, or rack.\u201d<\/p>\n I\/O reliability and redundancy \u201cReliability of fault prone I\/Os is tied to the physical implementation,\u201d said Vinci\u2019s Radhakrishnan. \u201cThe protocol may define how data moves, but the system still has to support that movement reliably under real power, thermal, mechanical, and manufacturing conditions.\u201d<\/p>\n Redundancy is essential. \u201cIn HPC, things that were nice to haves before, like silicon life cycle management, now are must-haves,\u201d said Rob Kruger, product management director for multi-die strategy and 3D IP at Synopsys. \u201cReliability is a key factor, and we follow OCP (Open Compute Project) standards for reliability, but we also add features such as redundant links in there.\u201d<\/p>\n Further, I\/Os can fail during assembly or in the field. \u201cSay you\u2019re doing a 3D link and there\u2019s a failed hybrid bond, which could be a problem during assembly. We have redundant links to replace those in the system,\u201d Kruger explained. \u201cThe same is true for UCIe links that connect to chiplets. You could have redundant links to repair broken links during manufacturing, or in the field five years from now. Software can monitor, test, and repair these links.\u201d<\/p>\n Telemetry plays a significant role here. \u201cYou might have sensors for process, voltage, and temperature, and signal integrity, for example. Then you aggregate that data and send it off to higher levels of the network,\u201d said Kruger. \u201cHow do you aggregate that data? Do you do that with software? Software is fine, but maybe you have thousands of I\/Os in the data center, all running software. You might choose a hardware-first approach instead, and the software is there as a backup. In that case, coordinating with the higher-level system is another challenge.\u201d<\/p>\n Clusters for giant models \u201cIf you see the evolution of computing, there was an era before 2012 where, within a processor, you had multiple cores integrated,\u201d said Priyank Shukla, director of product management for interface IP at Synopsys<\/a>. \u201cThen there was a time that within a rack or unit of a server, you had multiple processors, increasing the throughput. But after ChatGPT, we realized a lot of unstructured data can be processed with a different accelerator. You don\u2019t just need a processor. You need a workload-specific accelerator, and that needs to be connected. The scaling laws for large language models \u2014 not for CMOS \u2014 dictate that if you throw a large quantity of unstructured data at it, you get a very well-trained model. We are trying to create a large cluster that can act as a single, unified compute facility, which is different from what we are doing now. When the whole cluster has to act as a single computer, you need to provide memory to each compute node. You have to pass information across chips, and these offer different kinds of challenges for interconnects and I\/O, along with other components.\u201d<\/p>\n This passing of data is critical. \u201cWhen we say interconnect, we generally talk about the physical layer, or just one layer above,\u201d said Shukla. \u201cThese are very fundamental at the protocol level, as well. The idea here is you can pass data \u2014 not just simple data, but coherent memory data \u2014 to different nodes. There is no end to how much data.\u201d<\/p>\n A unified cluster can significantly boost performance. \u201cFrom an I\/O point of view, what it means is you have to escape as much as possible from one chip,\u201d Shukla explained. \u201cThe limitation is you can\u2019t stuff more compute within a chip, so two chips have to act together. The bottom line is how fast they can talk. The chip has a limited edge on the die, so you want to utilize as much bandwidth from a limited beachfront.\u201d<\/p>\n Beachfront density refers to the edge of the die and how much data can be transferred per millimeter. \u00a0\u201cIf you have an accelerator, the code is matrix multiplication or TensorFlow, but then the accelerator needs to talk with others, so they need an I\/O, and that\u2019s why people say chiplet I\/O,\u201d explained Shukla. \u201cBut how do you integrate it? There are different considerations. If you co-package them within a package and your I\/O die is at the top, it gets a chance to be at the faceplate of die. With liquid cooling, you can cool this die. But the base die, which is buried under it, doesn\u2019t have a path to extract heat, and that brings a different challenge.\u201d<\/p>\n Particularly in the AI space, these innovations are necessary because designers recognize that compute is not the only limitation to growth. \u201cData movement and memory access are really the problem within a chip, as well as across chips,\u201d Baya\u2019s Gayen said. \u201cHow are we able to make sure that these crazy, giant AI models, which have gigabytes and gigabytes of footprint, perform efficiently? This is where you get a lot more emphasis on the rack scale design that Nvidia has, for example, with the NVL72 exascale system.\u201d<\/p>\n NVL72 was a milestone to show connectivity is king, Gayen observed. \u201cHow do you move the data across the GPUs? The whole idea is that you don\u2019t want a single GPU \u2014 you want a ton of GPUs to act as if they\u2019re one giant GPU. That\u2019s where NVLink, and the NV switches associated with it, allowed Nvidia to build a comprehensive system that was not just hyper-focused on compute.\u201d<\/p>\n From a chip architect\u2019s perspective, clusters raise four practical concerns, according to Axiomise\u2019s Darbari:<\/p>\n Performance and scalability. The cluster\u2019s effective performance is a combination of local compute throughput and the cost of moving data and synchronizing work across nodes. High-performance clusters only scale well when the interconnect and software stack can keep the compute engines busy. Congestion challenges and specifications \u201cGPU clusters will process the data, and then at specific times, they exchange the results to traffic patterns called collective communication library (CCL), which generates huge amounts of traffic, requiring a huge number of ports at high speeds,\u201d said Razvan Arhip, product manager, AI and Network Test Solutions, at Keysight Technologies, in a recent webinar<\/a>.<\/p>\n With this traffic approach, designers need to avoid idle GPUs caused by network traffic bottlenecks. \u201cGPUs are expensive, and the clusters are expensive, so you don\u2019t want to idle them because of the network,\u201d said Arhip. \u201cYou have to have latencies very low and loss close to zero to avoid retransmissions, which eat up time. You can no longer rely on the upper protocols like TCP (Transmission Control Protocol) in the data centers to fix the loss. You need to minimize the congestion that is generating the loss, so you need to deal with the loss at as low a layer as you can, and this is why new congestion control mechanisms were adopted, like the DCQCN (Data Center Quantized Congestion Notification). This is also why the Ultra Ethernet Consortium released the Link Layer Retry (LLR), which performs retransmissions at layer two. These and CBFC (Credit-Based Flow Control) are driven by big companies in this space.\u201d<\/p>\n Finally, to mitigate network interconnect failures and associated I\/O congestion in large-scale AI training clusters, the OCP recently published an open-source Multipath Reliable Connection (MRC) protocol [1].<\/p>\n According to the related technical paper [2], \u201ca new RDMA (Remote Direct Memory Access)-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions.\u201d Further, it lowers latency, many more nodes are reachable in one hop, cost and power consumption are reduced, the impact of an in-network failure is much less, and it is possible to lose a NIC-T0 link without bringing down the training job.<\/p>\n Conclusion \u201cThese chips are not being built for an academic purpose,\u201d said Cadence\u2019s Khan. \u201cYou want to have the best performance, showcase great systems, sell a lot, and make a lot of money, so the tradeoffs are multi-pronged.\u201d<\/p>\n That means chip architects must weigh the options in terms of I\/Os and packaging. \u201cWhen you go back to even the simplest case of an SoC being disaggregated, there\u2019s a cost,\u201d Khan noted. \u201cYou add some latencies. You add power bonds at the interface. Now, is this going to fit in your budget, or are you better off with a monolithic solution?\u201d<\/p>\n The bottom line is there are many options. The challenge is to build a balanced system that optimizes whatever is most important for the end user. \u201cAt the end of the day, design teams are trying to solve a multi-dimensional problem, and each type of I\/O and advanced packaging has its own challenges and advantages.\u201d Synopsys\u2019 Shukla said.<\/p>\n References [2] Resilient AI Supercomputer Networking<\/a> using MRC and SRv6 (AMD, Broadcom, Intel, Microsoft, NVIDIA, and OpenAI)<\/p>\n Related Articles Swapping Out Chiplets: I\/Os Vs. Compute<\/a>
Numerous I\/O tradeoffs must be correctly balanced to ensure the commercial success of an AI chip. \u201cWhat tradeoff you make will have an impact on your airflow, cooling, rack design, the power coming into the rack, and so on,\u201d said Arif Khan, vice president of product management and marketing for the Silicon Solutions Group at Cadence<\/a>.<\/p>\n![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/io1.png\")
Fig. 1: AI scaling architectures with 1.6T Ultra Ethernet, UALink, and OSFP (Octal Small Form Factor Pluggable) I\/O connectors. Source<\/a>: Synopsys<\/p>\n
In high-performance computing systems, both I\/O subsystems and interconnects are significant sources of faults and performance degradation.<\/p>\n![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/io2.png\")
Fig. 2: A simplified data center network displaying connectivity required, with UCIe an option for 1.6T interconnects in I\/O chiplets for AI data centers Source<\/a>: Synopsys<\/p>\n
One of the problems the HPC ecosystem is trying to solve is how to make a whole cluster of compute nodes act as a single computer, with I\/Os a central consideration.<\/p>\n![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/io3.png\")
Fig. 3: AI connectivity in the data center. Source<\/a>: Synopsys<\/p>\n
\nLatency and tail behavior. Once the system is distributed, time-to-result depends not just on core and memory latency, but also on network hops, congestion, synchronization overhead, and queueing effects. Small delays can compound quickly at scale.
\nPower and energy. Power becomes a fleet-level property rather than a chip-level number. Choices around node design, accelerator mix, power management, and thermal limits directly affect rack density, cooling, and the practical economics of the cluster.
\nNetworking as part of the architecture. In a cluster, the network is not just plumbing. It is a first-class architectural element. Bandwidth, latency, topology, congestion behavior, and fault tolerance all shape what workloads scale well and what overall efficiency looks like, Darbari explained.<\/p>\n
AI era challenges include internet, cloud, and data traffic generated by video-on-demand and voice commands, along with new types of traffic from AI training data centers, as well as bursty traffic from AI inference.<\/p>\n
The competition to deliver accurate, lightning-fast AI capabilities across every sector is fierce. Every element of the HPC system or cluster is under pressure to perform optimally without failure.<\/p>\n
[1] Multipath Reliable Connection<\/a> (MRC) Specification (Open Compute Project)<\/p>\n
Confusion Grows With More Interconnect Options And Tradeoffs<\/a>
Each standard serves a specific use case, so chip architects are choosing more than one for a single design.<\/p>\n
Multi-die assemblies give chip architects the option to change some dies while keeping the rest of the system intact, but which is best to keep?<\/p>\n