The AI Infrastructure Stack: Four Layers Defined

Understanding AI research infrastructure requires moving beyond a simple equation of 'AI infrastructure = GPUs.' Research leaders who think only about compute capacity make systematically suboptimal infrastructure investments because they neglect the storage, networking, and software layers that determine whether compute capacity is actually usable for research. A complete AI infrastructure investment strategy addresses all four layers of the stack.

Compute Layer. The compute layer is the most visible and most capital-intensive. Three deployment models exist. On-premise GPU clusters provide dedicated research capacity with predictable cost structure and full data sovereignty. They are appropriate when utilization is high and data sensitivity is elevated. Cloud GPU instances (AWS EC2 P-family, Azure NDv-series, Google TPU/GPU instances) provide on-demand capacity that scales with demand without CapEx commitment: appropriate when utilization is variable or uncertain, when researchers need access to diverse hardware types, or when time to first experiment matters more than long-run cost optimization. NSF ACCESS (formerly XSEDE) provides federally funded national supercomputing capacity to research institutions via allocation processes, appropriate for compute-intensive workloads that justify the allocation application overhead and can tolerate some scheduling latency. Most research institutions benefit from a hybrid deployment model that uses on-premise capacity for predictable baseline workloads and cloud or ACCESS capacity for bursting during peak demand periods.

Storage Layer. The storage layer is frequently underinvested relative to compute. Effective AI research data management requires three storage tiers: high-performance parallel storage (GPFS, Lustre, or equivalent) for active training data and intermediate model checkpoints, where throughput rather than capacity is the critical metric; object storage (Ceph, MinIO, or commercial equivalents) for large dataset archiving and sharing, where cost per terabyte matters more than access latency; and archive storage (tape or cold cloud tiers) for long-term data retention required by funder data management plans. The ratio of storage capacity to compute capacity that enables productive research workflows varies by research domain, but a common reference point is 5 to 10 TB of high-performance storage per GPU node.

Networking Layer. The networking layer connects compute to storage and connects the research computing environment to the outside world. For GPU clusters training models that require inter-GPU communication (distributed training), InfiniBand interconnects between nodes are typically essential, Ethernet's latency characteristics produce prohibitive slowdowns for multi-node distributed training at scale. For external connectivity, high-speed links (10 Gbps or higher) to Internet2 or regional research networks enable data transfer with collaborators and access to cloud services. Insufficient networking investment is a common cause of compute underutilization: researchers often discover they cannot feed GPUs fast enough from storage, or cannot transfer large datasets from collaborators efficiently, creating bottlenecks that prevent the compute investment from being fully utilized.

Software Layer. The software layer encompasses container orchestration (Slurm for job scheduling on HPC environments, Kubernetes for cloud-native workloads), ML platforms (MLflow, Weights and Biases for experiment tracking; Hugging Face, PyTorch, TensorFlow ecosystems), and model registries for managing trained models across research groups. Software layer investments are often lower in CapEx than the other three layers but higher in ongoing operational effort: software environments must be maintained, updated, and supported across the research community. Centralized software environment management (containerized environments managed by research computing) reduces the per-researcher software maintenance burden and improves reproducibility.

Build, Buy, or Share: A Decision Framework with Crossover Analysis

Every AI infrastructure investment requires a decision about deployment model: build (on-premise, institution-owned), buy (purchase service through cloud or managed providers), or share (participate in a consortium or use national facilities). This is not a binary choice, most institutions deploy a hybrid across all three models, but each workload and each research community has characteristics that favor one model over others.

The Build Decision. Building on-premise infrastructure is favored when: utilization is expected to be high (typically above 60-70% sustained), data sensitivity requires on-premise processing (HIPAA, FERPA, classified), research workflows require very low storage-to-compute latency that cloud networking cannot provide, and the institution has the technical staff to manage and maintain the infrastructure. The key disadvantage of building is the CapEx commitment: a modest 64-GPU cluster typically costs $1.5M to $3M in hardware plus installation, and becomes partially obsolete within 3-5 years as GPU architectures advance. A build decision should be accompanied by a refresh plan that specifies how the infrastructure will be updated over its useful life.

The Buy Decision (Cloud). Cloud GPU instances are favored when: utilization is low or highly variable, researchers need access to diverse GPU types not available on-premise, time to first experiment is a research priority, or the institution lacks technical staff to manage on-premise infrastructure. The key disadvantage of cloud is cost at high utilization: cloud GPU costs can be 3-5x higher than on-premise amortized costs at sustained high utilization levels. Reserved instance pricing (1-year or 3-year commitments) reduces this premium significantly for predictable workloads.

The Share Decision (Consortia and NSF ACCESS). Shared infrastructure through a university consortium or national facilities is favored when: no single institution can justify the full infrastructure investment alone, the workload is large enough to require national-scale resources, or the research requires access to unique datasets or instruments co-located with national computing facilities. NSF ACCESS provides high-quality computing resources and technical support, but requires allocation applications that must be renewed, introduces scheduling variability that makes deadline-sensitive research workflows difficult, and has limited data privacy protections for sensitive research data.

TCO Crossover Analysis. The total cost of ownership (TCO) crossover analysis compares on-premise versus cloud costs over a 5-year horizon. The typical TCO crossover occurs at approximately 60-70% sustained utilization: below this threshold, cloud is more cost-effective because the on-premise CapEx is spread over fewer hours of use; above this threshold, on-premise is more cost-effective because the amortized hardware cost per GPU-hour falls below cloud spot or reserved rates. This crossover analysis should be updated annually as cloud pricing, GPU hardware costs, and institutional utilization patterns change.

Five-Year Total Cost of Ownership Modeling

Research leaders making AI infrastructure decisions require a credible five-year TCO model to evaluate investment options honestly and communicate cost implications to CFOs and boards. TCO modeling captures all costs across the full asset lifecycle, not just the initial purchase price.

On-Premise TCO Components. On-premise infrastructure TCO includes: hardware acquisition (GPU servers, storage arrays, networking switches, rack infrastructure), the largest single cost component; data center space and power (colocation costs or allocated institutional facility costs, including power draw at typical AI workload intensity, a modern GPU cluster can draw 10-20 kW per node); cooling (air cooling is becoming inadequate for high-density GPU deployments; liquid cooling systems add upfront cost but significantly reduce cooling energy costs); facilities and power make up 30-50% of 5-year on-premise TCO in many institutional contexts; personnel (research computing staff required to manage the infrastructure, typically 1 FTE per 50-100 GPU nodes for a well-managed system); maintenance and warranty (vendor support contracts, hardware replacement); and network bandwidth (data center interconnects, external connectivity).

Cloud TCO Components. Cloud GPU TCO includes: instance costs (on-demand rates are typically the most expensive option; reserved instances at 1-year or 3-year commitments reduce costs 30-50%; savings plans and committed use discounts offer additional flexibility); egress costs (data transfer out of cloud to on-premise or to collaborators is often the most surprising cost for research teams, large dataset transfers can cost tens of thousands of dollars in egress fees); software licensing (some ML platforms have per-instance licensing costs in cloud environments); and management overhead (researchers managing their own cloud accounts without institutional governance typically incur 20-40% waste from idle instances and misconfigured resources).

Hybrid Optimization. The optimal hybrid strategy for most research institutions uses on-premise infrastructure for the predictable baseline workload (typically 50-70% of annual compute demand) and cloud for burst workloads and specialized hardware requirements. This hybrid approach reduces both the CapEx commitment of a fully on-premise strategy and the per-unit cost of a fully cloud strategy. A key implementation detail is maintaining a unified job scheduling environment that can dispatch jobs to either on-premise or cloud resources based on queue depth, job priority, and cost policy. This enables researchers to access both resource types without managing separate environments.

Compute Allocation: Priority Queues, Fair-Share Scheduling, and Burst Access

Making an infrastructure investment decision is only the beginning. Research leaders must also establish governance policies for how shared compute resources are allocated across the research community. Poor allocation governance is a primary cause of researcher dissatisfaction with institutional AI infrastructure, regardless of how much compute has been purchased.

Priority Queue Systems. Most HPC job schedulers (Slurm is the most common in academic research computing environments) support multi-tier queue configurations that reflect institutional priorities. A typical research AI cluster configuration includes: a 'research' queue serving principal investigators and graduate students for research projects, with per-group allocations and preemption policies; a 'training' or 'coursework' queue serving instructional AI courses with smaller allocations and time limits appropriate for coursework-scale experiments; and a 'priority' or 'deadline' queue for urgent grant deliverables or conference paper submission deadlines, with access granted by the research computing director upon documented justification.

Fair-Share Scheduling. Fair-share scheduling algorithms (implemented in Slurm via its 'fairshare' mechanism) adjust each research group's job priority based on their historical usage relative to their allocated share. Research groups that have recently used more than their allocated share receive lower scheduling priority until their share balance recovers; groups that have used less than their allocated share receive higher priority. This mechanism prevents any single research group from monopolizing the cluster during high-demand periods without requiring manual administrative intervention.

Burst Access Policies. Research leaders should establish explicit policies for temporary burst access: situations where a research group needs dramatically more compute than their standing allocation for a limited period (submission deadlines, exploratory experiments for a new grant proposal, time-sensitive clinical research). Burst policies should specify: the conditions under which burst access is granted, the maximum burst multiplier (typically 2-3x the standing allocation), the maximum duration (typically one to two weeks), and the approval process (self-service with automatic approval up to some threshold; administrative review for larger requests). Clear burst policies reduce the frequency of emergency escalations to research computing leadership and create transparent expectations for research teams.

Course Allocation Pools vs. Research Pools. AI coursework presents a distinct allocation challenge: the demand is highly predictable and time-correlated (end of semester spikes, specific homework deadlines), but the per-student compute need is much smaller than typical research workloads. Maintaining separate allocation pools for course use versus research use prevents the two communities from competing for the same resources during high-demand periods. Course pools should be sized based on enrollment data and course compute requirements, and should include automatic scale-up mechanisms during final project periods.

Data Infrastructure: Research Data Management, FAIR Implementation, and Sensitive Data

AI research depends on data as much as it depends on compute. An institution that has invested in GPU infrastructure without addressing data infrastructure is like an institution that has built a powerful engine without a fuel system. Research data management (RDM) infrastructure is the fuel system for AI research, and it requires as much strategic investment as compute.

Research Data Management Plans as Funding Requirements. Since January 2023, NIH has required a Data Management and Sharing Plan (DMSP) for all NIH-funded research. NSF has long required data management plans. DOE, DOD, and other major funders are progressively tightening data management requirements. These requirements create both a compliance obligation and a strategic opportunity: institutions that develop strong RDM infrastructure, data repositories, deposit workflows, metadata standards, are better positioned to comply with funder requirements and to leverage their data assets for AI research.

Implementing FAIR Data Principles. FAIR (Findable, Accessible, Interoperable, Reusable) principles provide the operational framework for data management that supports AI research. Findable means data is registered or indexed in a searchable catalog with rich metadata. Accessible means data can be retrieved by standardized protocols with appropriate authentication and authorization. Interoperable means data uses broadly accepted formats and ontologies that enable combination with other datasets. Reusable means data is released with a clear usage license and sufficient context documentation. Implementing FAIR at the institutional level requires: a research data catalog (enabling search and discovery of institutional datasets), a data deposit workflow (enabling researchers to deposit new datasets with appropriate metadata), and a metadata standards committee (ensuring consistent cataloging practices across disciplines).

Sensitive Data Enclaves. Many high-value AI research datasets contain sensitive information: patient health records (HIPAA), student education records (FERPA), government data with handling restrictions. Standard research computing environments are not appropriate for sensitive data research. They lack the access controls, audit logging, and data handling procedures required by applicable regulations. Institutions with significant health AI or educational AI research programs should establish dedicated sensitive data enclaves: computing environments with physical and logical separation, stringent access controls, comprehensive audit trails, prohibition of data export, and documented incident response procedures. Sensitive data enclaves are expensive to build and operate, but they enable research that would otherwise be impossible, making them strategic assets for institutions with clinical or educational research strengths.

Talent Roles for AI Research Infrastructure: RSEs, MLOps, and Data Engineers

The most common infrastructure failure mode is not hardware. It is talent. Institutions that invest millions in GPU hardware but neglect the human capital required to manage, configure, and support that infrastructure consistently see their compute resources underutilized, their researchers frustrated, and their software environments degraded. The talent roles required for institutional AI research infrastructure are distinct from both traditional IT staff and research faculty.

Research Software Engineers (RSEs) vs. Research IT Staff. RSEs are technically sophisticated professionals who can both write production-quality code and understand the scientific context of research computing. They are distinct from traditional IT support staff, who focus on infrastructure management rather than software development, and from research faculty, who conduct original research rather than build research tools. RSEs support faculty by: optimizing code for GPU execution, building data pipelines that feed compute efficiently, containerizing research workflows for reproducibility, and developing experiment management tooling. The RSE role is increasingly recognized as a distinct career track in research computing, with professional organizations (US-RSE) developing standards and community practices.

MLOps Specialists. MLOps (Machine Learning Operations) specialists bridge the gap between ML research and production deployment. In a research context, MLOps specialists are responsible for: model registry management (tracking trained models across research groups), experiment tracking infrastructure (MLflow, Weights and Biases deployment and administration), continuous integration pipelines for research code, and the deployment infrastructure that takes research models into research applications (not commercial products). MLOps specialists can be deployed centrally (serving all research groups through a shared service) or embedded (dedicated to a single large research center or department). The choice depends on whether research computing needs are homogeneous enough for central service delivery or diverse enough to require specialized embedded support.

Data Engineers. Data engineers design and maintain the data pipelines that feed AI research systems with clean, structured, and appropriately governed data. They are responsible for: ETL (extract, transform, load) pipelines that move data from source systems to research compute environments, data quality monitoring that ensures model training data meets defined quality standards, schema management that maintains data structure consistency across updates, and integration of institutional data sources with external datasets from national repositories and partner institutions. Data engineers are one of the most critical and most scarce talent roles for institutions with ambitions to use institutional data (patient data, student records, administrative data) in AI research.

AI Research Librarians. An emerging and underappreciated role in the AI research talent ecosystem is the AI research librarian: a library professional with advanced skills in research data management, dataset discovery, data licensing, and AI research ethics. AI research librarians support faculty by: identifying relevant training datasets from national and international repositories, advising on data licensing and usage rights for AI training purposes, assisting with data management plan compliance, and providing guidance on responsible AI research practices. Institutions that embed AI research librarians in research data services create a significant capability advantage for faculty who cannot independently navigate the increasingly complex data licensing and governance landscape for AI research.

Grant-Funded Infrastructure: Allowable Costs, Equipment Management, and Indirect Cost Recovery

A significant portion of AI research infrastructure at most research institutions is funded through research grants. Understanding how different federal funders treat infrastructure costs, how equipment is managed across the grant lifecycle, and how indirect cost recovery works is essential for research leaders who want to maximize infrastructure investment while maintaining compliance.

Allowable Costs by Sponsor. Federal sponsors differ significantly in what infrastructure costs they consider allowable. NSF typically allows computing equipment as a direct cost in grants where the equipment is clearly necessary for the proposed research, but has specific requirements about equipment sharing and avoidance of unnecessary duplication. NIH's approach to computing equipment is more restrictive, equipment over $5,000 (the definition of 'equipment' under Uniform Guidance) can typically only be charged to grants when it is dedicated primarily to that grant's research and is not normally provided by the institution's research computing environment. DOD funding (including DARPA and AFRL) may allow infrastructure costs but often requires specific equipment use plans and may restrict equipment to specific facility locations. Understanding these differences is critical when designing research computing infrastructure for grant funding, an institution that charges computing infrastructure costs to NIH grants inappropriately may face audit findings and financial penalties.

Tracking Equipment Across Grants. Research equipment with unit cost above the federal equipment threshold ($5,000 under Uniform Guidance, though some sponsors use higher thresholds) must be tracked through the institution's property management system for the duration of its useful life. For AI research equipment (GPU servers, storage arrays) that is purchased on one grant but subsequently used across multiple projects and multiple grants, the tracking and allocation of equipment use time across grants requires proactive property management. Research leaders should work with their sponsored programs offices to establish equipment use logs, allocation methodologies, and disposition procedures that satisfy both the original funder's requirements and subsequent funder requirements if the equipment is shared across projects.

Indirect Cost Recovery. Indirect costs (also called F&A - Facilities and Administrative costs) represent the institution's overhead recovery from sponsored research. AI infrastructure investments have a complex relationship with indirect cost recovery. Infrastructure purchased as direct costs on grants generates no additional F&A recovery beyond what is included in the grant budget. Infrastructure purchased with institutional funds (or through cost sharing) is recovered through the institution's negotiated F&A rate, which includes a depreciation component for equipment. Research leaders should work with their sponsored programs and finance offices to understand how proposed AI infrastructure investments affect the institution's F&A rate calculation, large CapEx investments in computing infrastructure can significantly affect the rate in future years.

Infrastructure Sustainability: Power Efficiency, Carbon Footprint, and Environmental Responsibility

AI research infrastructure is one of the most energy-intensive activities in the modern research university. A dense GPU cluster can consume as much electricity as a medium-sized academic building. As institutions adopt sustainability commitments and as environmental accountability becomes a factor in research funder decisions and faculty recruitment, research leaders must treat the environmental footprint of AI infrastructure as a strategic consideration alongside cost and performance.

Power Usage Effectiveness (PUE). PUE is the standard metric for data center energy efficiency, defined as total data center power draw divided by IT equipment power draw. A PUE of 1.0 is theoretically perfect (all power goes to computing equipment), while higher values indicate increasing proportions of power used for cooling, lighting, and other overhead. Modern well-designed data centers targeting AI workloads should achieve PUE of 1.2 to 1.4. Legacy academic data centers with older cooling infrastructure often run at PUE of 1.5 to 2.0 or higher, meaning 50-100% overhead power for every unit of computing power. When evaluating on-premise AI infrastructure investments, the PUE of the proposed data center location is a critical operational cost factor, a high-PUE facility significantly increases the electricity cost of the same GPU workload.

Carbon Footprint and Scope 2 Emissions. AI GPU compute generates significant Scope 2 carbon emissions (emissions from purchased electricity). Reporting AI research carbon emissions is an emerging expectation in some research communities and is increasingly appearing in funder sustainability requirements. Calculating the carbon footprint of AI research requires knowing: the electricity consumption of the compute workload, the carbon intensity of the grid supplying power to the data center (which varies significantly by region and varies through the day), and any renewable energy certificates (RECs) purchased by the institution. Research computing centers should track GPU-hour energy consumption and provide estimated Scope 2 emissions data to researchers who need it for grant reporting or publication.

Liquid vs. Air Cooling. Traditional air cooling becomes increasingly inadequate for high-density GPU deployments as GPU thermal design power (TDP) increases with each GPU generation. Modern AI GPUs (NVIDIA H100, H200) have TDPs of 350-700W per GPU, well beyond what conventional air cooling can effectively manage at density. Direct liquid cooling (DLC) systems, where cooling fluid flows directly to GPU heat sinks, are becoming standard for high-density AI clusters. DLC systems have higher upfront installation costs but significantly lower PUE (1.05-1.15 is achievable with DLC vs. 1.2-1.5 for air cooling) and enable higher GPU density per rack. Research leaders planning GPU cluster investments should evaluate liquid cooling options as a long-term infrastructure choice rather than accepting the default air-cooling assumptions of legacy data center planning.

The AI Infrastructure Roadmap: Three Phases to National Competitiveness

AI research infrastructure investment requires a phased approach that matches capability building with the institution's evolving research program. Committing to a national-scale infrastructure investment before the research program and talent base are ready to utilize it produces the same underutilization problem as any infrastructure investment made ahead of institutional readiness. A three-phase infrastructure roadmap provides a structured path from initial capability to national competitiveness.

Phase 1: Pilot Infrastructure (0-12 months). The pilot phase establishes the minimum viable AI research computing environment while the institution builds the research program and talent base to support larger investments. Pilot infrastructure typically consists of: a small on-premise GPU node cluster (4-16 GPUs) or expanded cloud research computing access through institutional agreements with AWS, Google, or Microsoft; a basic research data management platform (institutional instance of OSF, Dataverse, or DSpace with AI-appropriate metadata schemas); a Slurm-based job scheduler with basic fair-share configuration; and a pilot sensitive data enclave if health or educational AI research is a strategic priority. The pilot phase should include explicit utilization monitoring: if the pilot infrastructure is consistently oversubscribed, it is a strong signal that expanding to Phase 2 is warranted; if it is consistently underutilized, the institution should investigate whether talent, data, or governance gaps are preventing adoption before investing in expansion.

Phase 2: Scalable Shared Infrastructure (12-36 months). The scalable infrastructure phase transitions from pilot to institutional-scale research computing. Key investments include: expansion of on-premise GPU capacity to the level justified by Phase 1 utilization evidence (typically 64-256 GPUs for a mid-sized research university), investment in high-performance parallel storage at a ratio appropriate to the GPU investment, establishment of an MLOps platform for experiment tracking and model registry, and expansion of the sensitive data enclave if pilot phase evidence supports significant sensitive data AI research demand. The scalable phase should also include the governance infrastructure that enables sustained shared use: fair-share scheduling implementation, allocation policies, user onboarding processes, and a research computing advisory committee that provides faculty input into allocation and priority decisions.

Phase 3: National Competitiveness (36-60 months). The national competitiveness phase positions the institution as a serious player in national AI research competitions: major federal AI institutes, large multi-institutional grants, and national leadership in chosen research focus areas. Infrastructure investments in this phase are calibrated to the requirements of the institution's strategic AI focus areas: a health AI focus may require expansion of the sensitive data enclave and integration with clinical data systems; a robotics AI focus may require specialized simulation infrastructure; a natural language processing focus may require large-scale storage for training corpora. Phase 3 infrastructure should be planned in conjunction with specific grant proposals and partnership development efforts, so that the infrastructure investment is justified by concrete research program commitments rather than aspirational capacity building.