Frequently Asked Questions
As we talk with potential FABRIC collaborators, we are getting a lot of wonderful questions. We've created a list of questions that we hear often to help you navigate your own FABRIC curiosities.
FABRIC will allow experimenters to instantiate compute instances in the form of bare-metal servers, VMs and containers with significant compute and storage capabilities as well as directly attached PCI devices: multi-port SmartNICs, GPUs and FPGAs. These compute instances will be interconnected with on-demand high-bandwidth links up to 25-100Gbps in the Core and higher bit rates in the SuperCore portion of the topology. These compute instances can also have storage in the form of dedicated NVMe drives and slower, but larger dynamically partitionable block storage. Several sites will provide on-demand access to 100Gbps P4-capable switches, which can similarly be linked into experimenter topologies. Compute instances will be accessible by experimenters via SSH, other more scalable experiment control tools may be contributed by the community later. FABRIC will offer different network services: a best-effort Layer 2/IPv6 network matching FABRIC footprint, on-demand Layer 2 and Layer 3 topologies and peering between slice topologies and production networks. Public cloud peering at 100Gbps in two bi-coastal locations will be offered through a partnership with Internet2.
While FABRIC takes advantage of experience and approaches to building widely distributed testbeds collected from GENI, there are differences between FABRIC and GENI that drastically change and expand the types of experiments that can be performed:
- FABRIC has a programmable core network infrastructure
- FABRIC provides guaranteed quality of service by utilizing its own dedicated optical infrastructure or relying on dedicated L2 capacity wherever possible to create QoS-guaranteed connections.
- FABRIC provides access to a variety of programmable PCI devices Network cards, GPUs, FPGAs
- FABRIC supports highly accurate timestamping of events using GPS-based clocks.
- FABRIC interconnects a large number of existing scientific, computational and experimental facilities
- FABRIC experimenter network topologies can peer with production networks on-demand
- FABRIC supports a broader range of data transfer speeds (10Gbs, 25Gbps, 100Gbps, 1000Gbps) along with the ability to compute on data at these line speeds within the network.
A FABRIC Portal will provide basic browser-based access to FABRIC resources. Programmatic access will be provided with help of FABRIC client libraries that can be operated as a CLI or embedded into experimenter code. Initially Python bindings will be provided which later may be extended to other languages. FABRIC will provide a hosted Jupyter Hub environment for experimenters, which will come with Jupter notebooks automatically preloaded with FABRIC libraries and CLI tools. The portal and Jupyter Hub will use federated identity through CI Logon and support any institutional identity provider that is part of InCommon R&S (Research and Scholarship) federation. This includes the majority of US educational institutions.
Initial FABRIC experiments during the construction phase (2019-2023) will be done through individual agreements between specific researchers identified as early experimenters, and the FABRIC team. When FABRIC enters production it will have an AUP and an allocation policy that balances the needs of various experimenter groups (long-running and resource-intensive) and educational uses. Mechanisms will be put in place to enforce those allocation policies across the FABRIC footprint and will provide flexibility for FABRIC expansion site owners and Facility Partners to allow them to enforce site-specific allocation policies.
Current FABRIC designs envision supporting Mellanox ConnectX family of dual-port network cards, NVIDIA GPUs (T4s and RTX6000s) and Xilinx Alveo family FPGAs. FABRIC expects to provide limited support for GPU and FPGA application development workflows, instead partnering with other testbeds that offer similar devices so they can be used for developing the application codes that are then deployed in FABRIC.
Yes, FABRIC will support expansion sites (e.g. universities) that purchase their own FABRIC node (‘hank’) based on standard configurations that will be made available to the community, FABRIC team will also help facilitate connecting expansion sites to FABRIC topology either via Internet2 or with dedicated Layer2 or optical connectivity. We expect standard FABRIC Site BOMs to be available in the fall of 2020. Stay in touch by completing the Get Involved form.
FABRIC nodes also support a Bring Your Own Equipment (BYOE) model of contributions. Contributions of hardware to FABRIC can take a number of forms:
- PCI devices to be included in the available servers
- Network switching equipment added to individual sites
- Discrete servers added to individual sites
BYOE contributions are subject to a number of constraints:
- Available rack space at individual sites
- Available power (core sites are limited to only 4kW total power budget)
- Available PCI slots in servers
- Available ports on FABRIC node dataplane and management switches
Contributed hardware will be linked into the FABRIC management plane and individual resources become provisionable via FABRIC CF. More details on BYOE support can be found in the FABRIC Node Design document.
The FABRIC Measurement Framework is designed to be adaptable/programmable, scalable, extensible, and shareable. It’s goal is to support an automated instrumentation of experiments, enabling experimenters to collect, store, and publish detailed measurement data from users and the system/experiment. Examples include CPU, disk and network utilization, packet error counts and other interface statistics, packet capture traces with precise timestamps from inside the experiment topologies. FABRIC will also allow users to collect measurements and error statistics from the underlying optical/DWDM infrastructure and electrical power consumption data on a per-server basis. The framework is being realized using a measurement bus infrastructure based on pub/sub technology with efficient filtering, searching, and processing of measurement data within the testbed using a combination of testbed resources (e.g. GPUs and FPGAs for real-time training and inference on the data) and off-line processing outside the testbed using experimenter’s institutional or public cloud resources. The framework will allow for fine-grained precise measurements leveraging a highly-accurate PTP timing signal from a GPS receiver located at every FABRIC site. This will enable precise timestamping of packets using NIC cards (a.k.a., PacketGPS) with the accuracy of 10s of milliseconds. FABRIC will also allow for the collection of high-speed packet traces using RAM or high-speed NVMe drives at each site, which can be attached to individual experimenter compute instances. Larger data sets can be stored within experiment-attached block storage. The facility will also have between ½ and 1PB of object storage space split into 2 bi-coastal locations to help ease moving measurement and other experiment-related data in- and out- of experiments in programmatic fashion. FABRIC will collect logging and metrics from the underlying infrastructure to be made available to operators and experimenters. Additional details about the types of available data and data sharing policies for the facility will be devised in consultations with and shared with the community.
Yes, we definitely are interested. While our currently available funding does not support deploying hardware outside the US, we will be working with other NSF programs and NSF partners to make this possible. We are interested in collaborating with international partners to deploy exchange points that will allow them to interconnect their facilities to FABRIC. We will share our software and hardware designs, so that experimental facilities like FABRIC can emerge in other countries and be federated with us to create a world-wide 'testbed-of-testbeds’. We will be present at many international meetings where details of federation and connectivity are discussed so we can include them in our plans. If you are interested in collaborating internationally with us, please complete the Get Involved form at our website.
FABRIC is a scientific instrument for supporting research in cyber-infrastructure. This implies that its primary use is for experiments in a variety of areas related to internet architectures, cyber-security, distributed protocols and applications. Many of these experiments may need to be long-running and carry production-like traffic in order to truly understand the tradeoffs in the system that is being experimented on. However, by virtue of being experiments, they can’t be expected to provide production-grade reliability and robustness.
In order to support the widest possible variety of experiments spanning from blue-sky future Internet ideas to support for novel production science workflows, FABRIC will provide on-demand connectivity with production networks and facilities, to allow experimenters mix and match production and experimental resources in combinations that make sense for their experiments.
What is FABRIC?
Below are two resources—a presentation and a webinar—that may provide further information to assist with answering your FABRIC questions.