%% Generated by Sphinx. \def\sphinxdocclass{report} \documentclass[a4paper,11pt,english]{sphinxmanual} \ifdefined\pdfpxdimen \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen \fi \sphinxpxdimen=.75bp\relax \ifdefined\pdfimageresolution \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax \fi %% let collapsible pdf bookmarks panel have high depth per default \PassOptionsToPackage{bookmarksdepth=5}{hyperref} \PassOptionsToPackage{warn}{textcomp} \usepackage[utf8]{inputenc} \ifdefined\DeclareUnicodeCharacter % support both utf8 and utf8x syntaxes \ifdefined\DeclareUnicodeCharacterAsOptional \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}} \else \let\sphinxDUC\DeclareUnicodeCharacter \fi \sphinxDUC{00A0}{\nobreakspace} \sphinxDUC{2500}{\sphinxunichar{2500}} \sphinxDUC{2502}{\sphinxunichar{2502}} \sphinxDUC{2514}{\sphinxunichar{2514}} \sphinxDUC{251C}{\sphinxunichar{251C}} \sphinxDUC{2572}{\textbackslash} \fi \usepackage{cmap} \usepackage[LGR,T1]{fontenc} \usepackage{amsmath,amssymb,amstext} \usepackage{babel} \usepackage{substitutefont} \usepackage{tgtermes} \usepackage{tgheros} \renewcommand{\ttdefault}{txtt} \expandafter\ifx\csname T@LGR\endcsname\relax \else % LGR was declared as font encoding \substitutefont{LGR}{\rmdefault}{cmr} \substitutefont{LGR}{\sfdefault}{cmss} \substitutefont{LGR}{\ttdefault}{cmtt} \fi \expandafter\ifx\csname T@X2\endcsname\relax \expandafter\ifx\csname T@T2A\endcsname\relax \else % T2A was declared as font encoding \substitutefont{T2A}{\rmdefault}{cmr} \substitutefont{T2A}{\sfdefault}{cmss} \substitutefont{T2A}{\ttdefault}{cmtt} \fi \else % X2 was declared as font encoding \substitutefont{X2}{\rmdefault}{cmr} \substitutefont{X2}{\sfdefault}{cmss} \substitutefont{X2}{\ttdefault}{cmtt} \fi \usepackage{textalpha} \usepackage[Bjarne]{fncychap} \usepackage[,numfigreset=1,mathnumfig]{sphinx} \fvset{fontsize=auto} \usepackage{geometry} % Include hyperref last. \usepackage{hyperref} % Fix anchor placement for figures with captions. \usepackage{hypcap}% it must be loaded after hyperref. % Set up styles of URL: it should be placed after hyperref. \urlstyle{same} \addto\captionsenglish{\renewcommand{\contentsname}{Table of Contents}} \usepackage{sphinxmessages} \setcounter{tocdepth}{1} \title{Private 5G: A Systems Approach} \date{Oct 18, 2023} \release{Version 1.1\sphinxhyphen{}dev} \author{Peterson, Sunay, and Davie } \newcommand{\sphinxlogo}{\vbox{}} \renewcommand{\releasename}{Release} \makeindex \begin{document} \pagestyle{empty} \sphinxmaketitle \pagestyle{plain} \sphinxtableofcontents \pagestyle{normal} \phantomsection\label{\detokenize{index::doc}} \chapter*{Foreword} \label{\detokenize{foreword:foreword}}\label{\detokenize{foreword::doc}} \sphinxAtStartPar When I was a graduate student studying parallel computing in the early 1990s, the World Wide Web exploded onto the scene, and I wanted to understand how the Internet made such rapid advances possible. I picked up a copy of the first edition of \sphinxstyleemphasis{Computer Networks: A Systems Approach}, and I started reading. I was delighted to learn not only the Internet’s history and main concepts, but also to see examples of real code that illustrated how the protocols actually worked and how they could be changed. I was hooked. I loved the idea of working in the computer networking field, where the technologies we build can help bring people together and lower the barriers to innovation so that anyone who can write software can contribute. \sphinxAtStartPar As time went on, I saw that, while the Internet enabled rapid advances in end\sphinxhyphen{}host applications, the inside of the Internet was harder to change. Network devices were closed and proprietary, and protocol standards evolved slowly and painfully. In response, enabling innovation inside the network became a passion of mine, and of many other technologists eager to make the Internet better\textendash{}more secure, performant, reliable, cost\sphinxhyphen{}effective, and easier to manage. Gradually, the computer networking field changed, with the advent of software\sphinxhyphen{}defined networking, giving network owners\textendash{}such as the hyperscalers running large data centers\textendash{}much more control over the software inside their networks. (See \sphinxstyleemphasis{Software\sphinxhyphen{}Defined Networks: A Systems Approach} for more of this story!) Nowadays, computer networking really is about software. It’s a welcome change. \sphinxAtStartPar In recent years, the excitement has moved to the network edge with the emergence of 5G cellular access networks. No longer are wireless communication, computer networking, and cloud computing siloed parts of some larger ecosystem. They are brought together, allowing wireless devices to connect to nearby computing resources. Far beyond providing mobile Internet access, these networks enable exciting real\sphinxhyphen{}time applications like augmented and virtual reality (AR/VR), Internet of Things (IoT), self\sphinxhyphen{}driving cars, drones, robotic control, and more. Plus, these networks are not only the purview of large carriers that must invest significant resources in wireless spectrum, cell towers, and more. Rather, individual enterprises, campuses, and communities are deploying private 5G within their own organizations to support their own applications, using lightly regulated spectrum like CBRS (Citizens Broadband Radio Service). Opportunities for innovation abound! \sphinxAtStartPar However, now there is so much more to learn, to understand the many parts of 5G access networks as a single coherent system. The learning curve can be steep. Even fairly technical people usually know one or at most two parts of the system, and do not know how the parts relate to the larger whole. For example, I knew a good amount about computer networking, a little about cloud computing, and not much about wireless communication. \sphinxstyleemphasis{Private 5G: A Systems Approach}\textendash{}this book\textendash{}changed that. I learned what I needed to know about the radio access network, and how the pieces come together to be more than the sum of their parts. More than that, the book shows that, despite the complexity of the 3GPP cellular standards and the long and frustrating history of closed network equipment in the cellular networking space, open source software is gaining a foothold. Open source platforms like Aether and Magma\textendash{}both discussed in this book\textendash{}are seeing practical deployment in a wide variety of settings. The book even has a guide for readers to bring up the Aether platform, including a 5G small cell. \sphinxAtStartPar For me, then, the story comes full circle. Private 5G networks are something you can touch, code, and deploy yourself. Armed with the knowledge of how 5G access networks work, and with hands\sphinxhyphen{}on experience with open source software, just imagine the places you’ll go! \begin{DUlineblock}{0em} \item[] Jennifer Rexford \item[] Princeton, New Jersey \end{DUlineblock} \chapter*{Preface} \label{\detokenize{preface:preface}}\label{\detokenize{preface::doc}} \sphinxAtStartPar When we wrote our introductory 5G book three years ago, our goal was to help people with experience building Internet and cloud services to understand the opportunity to bring best practices from those systems to the mobile cellular network. On paper (and in the press) 5G had set an ambitious goal of transformative changes, adopting a cloud\sphinxhyphen{}inspired architecture and supporting a new set of innovative services. But the gap between that aspirational story and the reality of 40 years of network operators and hardware vendors protecting their incumbent advantages made for a challenging pivot. So we started with the basics, and set out to explain the fundamental networking concepts and design principles behind the myriad of acronyms that dominate mobile cellular networking. \sphinxAtStartPar Because 5G adopts many of the principles of cloud native systems, it promises to bring the feature velocity of the cloud to Telco environments. That promise is being delivered most successfully in private 5G deployments that are less constrained by existing Telco organizations and legacy infrastructure. What started out as sketches on a whiteboard three years ago is now becoming a reality: Several cloud providers are offering private 5G solutions for enterprises, and there is a complete open source implementation of a 5G\sphinxhyphen{}enabled edge cloud that the Internet community can learn from and build upon. \sphinxAtStartPar The architecture described in this book is not limited to private deployments. It includes the necessary background information about the mobile cellular network, much of which is rooted in its origin story as a Telco voice network, but the overarching theme is to describe the network through the lens of private deployments of 5G connectivity as a managed cloud service. This includes adopting best practices in horizontally scalable microservices, Software\sphinxhyphen{}Defined Networking (SDN), and cloud operational practices such as DevOps. These practices are appropriate for traditional operators, cloud providers, and enterprises alike, but it is emerging use cases in private deployments that will benefit first. \sphinxAtStartPar The book makes extensive use of open source software—specifically, the Aether and Magma projects—to illustrate how Private 5G can be realized in practice. The availability of open software informs our understanding of what has historically been a proprietary and opaque system. The result complements the low\sphinxhyphen{}level engineering documents that are available online (and to which we provide links) with an architectural roadmap for anyone trying to understand all the moving parts, how they fit together, and how they can be operationalized. And once you’re done reading the book, we encourage you to jump into the hands\sphinxhyphen{}on appendix that walks you through the step\sphinxhyphen{}by\sphinxhyphen{}step process of deploying that software in your own local computing environment. \section*{Acknowledgements} \label{\detokenize{preface:acknowledgements}} \sphinxAtStartPar The software described in this book is due to the hard work of the ONF engineering team, the Magma engineering team, and the open source communities that work with them. Bilal Saleem did the heavy lifting on Aether OnRamp (described in the \sphinxstyleemphasis{About the Software} Appendix), with a special thanks to Ajay Thakur, Andy Bavier, Gabriel Arrobo, and Muhammad Shahbaz for their guidance and feedback. \sphinxAtStartPar Thanks to the members of the community who contributed text or corrections to the book, including: \begin{itemize} \item {} \sphinxAtStartPar Edmar Candeia Gurjão \item {} \sphinxAtStartPar Muhammad Shahbaz \item {} \sphinxAtStartPar Mugahed Izzeldin \item {} \sphinxAtStartPar Robert MacDavid \item {} \sphinxAtStartPar Simon Leinen \item {} \sphinxAtStartPar Tiago Barros \item {} \sphinxAtStartPar Gabriel Arrobo \item {} \sphinxAtStartPar Ajay Thakur \end{itemize} \sphinxAtStartPar The picture of a Magma deployment in Chapter 5 was provided by Shaddi Hasan. \begin{DUlineblock}{0em} \item[] Larry Peterson, Oguz Sunay, and Bruce Davie \item[] April 2023 \end{DUlineblock} \chapter{Chapter 1: Introduction} \label{\detokenize{intro:chapter-1-introduction}}\label{\detokenize{intro::doc}} \sphinxAtStartPar Mobile networks, which have a 40\sphinxhyphen{}year history that parallels the Internet’s, have undergone significant change. The first two generations supported voice and then text, with 3G defining the transition to broadband access, supporting data rates measured in hundreds of kilobits per second. Today, the industry is transitioning from 4G (with data rates typically measured in the few megabits per second) to 5G, with the promise of a tenfold increase in data rates. \sphinxAtStartPar But 5G is about much more than increased bandwidth. 5G represents a fundamental rearchitecting of the access network in a way that leverages several key technology trends and sets it on a path to enable much greater innovation. In the same way that 3G defined the transition from voice to broadband, 5G’s promise is primarily about the transition from a single access service (broadband connectivity) to a richer collection of edge services and devices. 5G is expected to provide support for immersive user interfaces (e.g., Augmented Reality, Virtual Reality), mission\sphinxhyphen{}critical applications (e.g., public safety, autonomous vehicles), and the Internet of Things (IoT). Because these use cases will include everything from home appliances to industrial robots to self\sphinxhyphen{}driving cars, 5G will support not only humans accessing the Internet from their smartphones, but also swarms of autonomous devices working together on their behalf. \sphinxAtStartPar There is more to supporting these services than just improving bandwidth or latency to individual users. As we will see, a fundamentally different edge network architecture is required. The requirements for this architecture are ambitious, and can be illustrated by three classes of capabilities: \begin{itemize} \item {} \sphinxAtStartPar To support \sphinxstyleemphasis{Massive Internet of Things}, potentially including devices with ultra\sphinxhyphen{}low energy (10+ years of battery life), ultra\sphinxhyphen{}low complexity (10s of bits per second), and ultra\sphinxhyphen{}high density (1 million nodes per square kilometer). \item {} \sphinxAtStartPar To support \sphinxstyleemphasis{Mission\sphinxhyphen{}Critical Control}, potentially including ultra\sphinxhyphen{}high availability (greater than 99.999\% or “five nines”), ultra\sphinxhyphen{}low latency (as low as 1 ms), and extreme mobility (up to 100 km/h). \item {} \sphinxAtStartPar To support \sphinxstyleemphasis{Enhanced Mobile Broadband}, potentially including extreme data rates (multi\sphinxhyphen{}Gbps peak, 100+ Mbps sustained) and extreme capacity (10 Tbps of aggregate throughput per square kilometer). \end{itemize} \sphinxAtStartPar These targets will certainly not be met overnight, but that’s in keeping with each generation of the mobile network being a decade\sphinxhyphen{}long endeavor. \sphinxAtStartPar On top of these quantitative improvements to the capabilities of the access network, 5G is being viewed as a chance for building a platform to support innovation. Whereas prior access networks were generally optimized for known services (such as voice calls and SMS), the Internet has been hugely successful in large part because it supported a wide range of applications that were not even thought of when it was first designed. The 5G network is designed with this same goal: enabling future applications beyond those we fully recognize today. For an example of the grand vision for 5G, see the whitepaper from one of the industry leaders. \phantomsection\label{\detokenize{intro:reading-vision}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar Qualcomm Whitepaper. \sphinxhref{https://www.qualcomm.com/media/documents/files/whitepaper-making-5g-nr-a-reality.pdf}{Making 5G NR a Reality}. December 2016. \end{sphinxadmonition} \sphinxAtStartPar The 5G mobile network, because it is on an evolutionary path and not a point solution, includes standardized specifications, a range of implementation choices, and a long list of aspirational goals. Because this leaves so much room for interpretation, our approach to describing 5G is grounded in three mutually supportive principles. The first is to apply a \sphinxstyleemphasis{systems lens}, which is to say, we explain the sequence of design decisions that lead to a solution rather than fall back on enumerating the overwhelming number of acronyms or individual point technologies as a \sphinxstyleemphasis{fait accompli}. The second is to aggressively disaggregate the system. Building a disaggregated, virtualized, and software\sphinxhyphen{}defined 5G access network is the direction the industry is already headed (for good technical and business reasons), but breaking the 5G network down into its elemental components is also the best way to explain how 5G works. \sphinxAtStartPar The third principle is to illustrate how 5G can be realized in practice by drawing on specific engineering decisions made in an open source implementation. This implementation leverages best practices in building cloud apps, which is an essential aspect of 5G evolving into a platform for new services. This implementation also targets enterprises that are increasingly deploying 5G locally, and using it to help automate their manufacturing, retail, and business practices—a trend that has been dubbed \sphinxstyleemphasis{Industry 4.0}. Such enterprise\sphinxhyphen{}level deployments are known as \sphinxstyleemphasis{Private 5G}, but there is nothing about the technical approach that couldn’t be adopted throughout the more traditional “public mobile network” that comes to mind when you think about your cell service today. The only difference is that private deployments are more aggressively embracing the cloud practices that will ultimately distinguish 5G from earlier generations. \phantomsection\label{\detokenize{intro:reading-industry4-0}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar K. Schwab. \sphinxhref{https://www.weforum.org/about/the-fourth-industrial-revolution-by-klaus-schwab}{The Fourth Industrial Revolution}. World Economic Forum. \end{sphinxadmonition} \sphinxAtStartPar What this all means is that there is no simple definition of 5G, any more than there is for the Internet. It is a complex and evolving system, constrained by a set of standards that purposely give all the stakeholders many degrees of freedom. In the chapters that follow, it should be clear from the context whether we are talking about \sphinxstyleemphasis{standards} (what everyone must do to interoperate), \sphinxstyleemphasis{trends} (where the industry seems to be headed), or \sphinxstyleemphasis{implementation choices} (examples to make the discussion more concrete). By adopting a systems perspective throughout, our intent is to describe 5G in a way that helps the reader navigate this rich and rapidly evolving system. \section{1.1 Standardization Landscape} \label{\detokenize{intro:standardization-landscape}} \sphinxAtStartPar As of 3G, the generational designation corresponds to a standard defined by the \sphinxstyleemphasis{3rd Generation Partnership Project (3GPP)}. Even though its name has “3G” in it, the 3GPP continues to define the standards for 4G, 5G, and so on, each of which corresponds to a sequence of releases of the standard. Release 15 is considered the demarcation point between 4G and 5G, with Release 17 having been completed in 2022. \sphinxAtStartPar In addition to 3GPP\sphinxhyphen{}defined standards, national governments establish how the radio spectrum is used locally. Unlike Wi\sphinxhyphen{}Fi, for which there is international agreement that permits anyone to use a channel at either 2.4 or 5 GHz (these are unlicensed bands), governments have auctioned off and licensed exclusive use of various frequency bands to service providers, who in turn sell mobile access service to their subscribers. The use of licensed spectrum brings certain benefits such as greater control over the quality of service delivered, while also imposing costs both in terms of paying for licenses and in the complexity of the systems needed to manage access to the spectrum. We will explore how these costs and benefits play out in subsequent chapters. \sphinxAtStartPar There is also a shared\sphinxhyphen{}license band at 3.5 GHz, called \sphinxstyleemphasis{Citizens Broadband Radio Service (CBRS)}, set aside in North America for cellular use. Similar spectrum is being set aside in other countries. The CBRS band allows 3 tiers of users to share the spectrum: first right of use goes to the original owners of this spectrum (naval radars and satellite ground stations); followed by priority users who receive this right over 10MHz bands for three years via regional auctions; and finally the rest of the population, who can access and utilize a portion of this band as long as they first check with a central database of registered users. CBRS, along with standardization efforts to extend mobile cellular networks to operate in the unlicensed bands, opens the door for private cellular networks similar to Wi\sphinxhyphen{}Fi. This is proving especially attractive to enterprises looking to establish a \sphinxstyleemphasis{Private 5G} service. \sphinxAtStartPar The specific frequency bands that are licensed for cellular networks vary around the world, and are complicated by the fact that network operators often simultaneously support both old/legacy technologies and new/next\sphinxhyphen{}generation technologies, each of which occupies a different frequency band. The high\sphinxhyphen{}level summary is that traditional cellular technologies range from 700\sphinxhyphen{}2400 MHz, with new mid\sphinxhyphen{}spectrum allocations now happening at 6 GHz, and millimeter\sphinxhyphen{}wave (mmWave) allocations opening above 24 GHz. \sphinxAtStartPar While the specific frequency band is not directly relevant to understanding 5G from an architectural perspective, it does impact the physical\sphinxhyphen{}layer components, which in turn has indirect ramifications on the overall 5G system. We identify and explain these ramifications in later chapters, keeping in mind that ensuring the allocated spectrum is used \sphinxstyleemphasis{efficiently} is a critical design goal. \sphinxAtStartPar Finally, in addition to the long\sphinxhyphen{}established 3GPP standards body and the set of national regulatory agencies around the world, a new organization—called the \sphinxstyleemphasis{Open\sphinxhyphen{}RAN Alliance (O\sphinxhyphen{}RAN)} —has recently been established to focus on “opening up the Radio Access Network”. We’ll see specifically what this means and how the O\sphinxhyphen{}RAN differs from the 3GPP in Chapter 4, but for now, its existence highlights an important dynamic in the industry: 3GPP has become a vendor\sphinxhyphen{}dominated organization, with network operators (AT\&T and China Mobile were the founding members) creating O\sphinxhyphen{}RAN to break vendor lock\sphinxhyphen{}in. \section{1.2 Access Networks} \label{\detokenize{intro:access-networks}} \sphinxAtStartPar The mobile cellular network is part of the access network that implements the Internet’s so\sphinxhyphen{}called \sphinxstyleemphasis{last mile}. (Another common access technology is \sphinxstyleemphasis{Passive Optical Networks (PON)}, colloquially known as Fiber\sphinxhyphen{}to\sphinxhyphen{}the\sphinxhyphen{}Home.) These mobile access networks have historically been provided by both big and small \sphinxstyleemphasis{Mobile Network Operators (MNOs)}. Global MNOs such as AT\&T run access networks at thousands of aggregation points of presence across a country such as the US, along with a national backbone that interconnects those sites. Small regional and municipal MNOs might run an access network with one or two points of presence, and then connect to the rest of the Internet through some large operator’s backbone. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide1}.png} \caption{A global mobile network built by first aggregating traffic from hundreds of wireless base stations, and then interconnecting those aggregation points over the Internet.}\label{\detokenize{intro:id3}}\label{\detokenize{intro:fig-global}}\end{figure} \sphinxAtStartPar As illustrated in \hyperref[\detokenize{intro:fig-global}]{Figure \ref{\detokenize{intro:fig-global}}}, access networks are physically anchored at thousands of aggregation points of presence within close proximity to end users, each of which serves anywhere from 1,000\sphinxhyphen{}100,000 subscribers, depending on population density. In practice, the physical deployment of these “edge” locations vary from operator to operator, but one possible scenario is to anchor both the cellular and wireline access networks in Telco \sphinxstyleemphasis{Central Offices}. \sphinxAtStartPar Historically, the Central Office—officially known as the \sphinxstyleemphasis{PSTN (Public Switched Telephone Network) Central Office}—anchored wired access (both telephony and broadband), while the cellular network evolved independently by deploying a parallel set of \sphinxstyleemphasis{Mobile Telephone Switching Offices (MTSO)}. Each MTSO serves as a \sphinxstyleemphasis{mobile aggregation} point for the set of cell towers in a given geographic area. For our purposes, the important idea is that such aggregation points exist, and it is reasonable to think of them as defining the edge of an operator\sphinxhyphen{}managed access network. For simplicity, we sometimes use the term “Central Office” as a synonym for both types of edge sites. \sphinxAtStartPar Finally, one aspect of the mobile network that may not be obvious from \hyperref[\detokenize{intro:fig-global}]{Figure \ref{\detokenize{intro:fig-global}}} is that it supports global connectivity, independent of the Internet (which is technically just one of many available backbone technologies). That is, the cellular network supports a universal addressing scheme, similar in principle (but significantly different in details) from the Internet’s universal IP\sphinxhyphen{}based addressing scheme. This addressing scheme makes it possible to establish a voice call between any two cell phones, but of course, IP addresses still come into play when trying to establish a data (broadband) connection to/from a cell phone or other mobile device. Understanding the relationship between mobile addresses and IP addresses is a topic we will explore in later chapters. \section{1.3 Managed Cloud Service} \label{\detokenize{intro:managed-cloud-service}} \sphinxAtStartPar The previous section gives a decidedly Telco\sphinxhyphen{}centric view of the mobile cellular network, which makes sense because Telcos have been the dominant MNOs for the past 40+ years. But with 5G’s focus on broadening the set of services it supports, and embracing general platforms that can host yet\sphinxhyphen{}to\sphinxhyphen{}be\sphinxhyphen{}invented applications, the mobile cellular network is starting to blur the line between the access network and the cloud. \begin{sphinxShadowBox} \sphinxstylesidebartitle{5G, Wi\sphinxhyphen{}Fi, and the Role of Spectrum} \sphinxAtStartPar \sphinxstyleemphasis{WiFi networks use unlicensed radio spectrum that do not require WiFi network operators to get advance regulatory approval. At the same time, anyone can access the same spectrum, subject to limits on transmission power. As a result, WiFi networks share their bands with devices including baby monitors, cordless phones, etc., so the WiFi MAC layer assumes the presence of physical\sphinxhyphen{}layer interference. Enterprise WiFi deployments, such as those on college campuses and in corporate office buildings, perform more centralized management of interference across multiple overlapping access points, but risk of interference remains and thus the service remains best\sphinxhyphen{}effort.} \sphinxAtStartPar \sphinxstyleemphasis{Cellular access networks typically use licensed spectrum that is owned or leased by the carrier for long periods of time at high cost. Even “lightly licensed” spectrum such as CBRS offers more control over interference than Wi\sphinxhyphen{}Fi. Since the cellular radio has exclusive access to spectrum over a geographic region, cellular waveforms are designed for wide\sphinxhyphen{}area coverage and high spectral efficiency. Managing access to the spectrum, as we shall see, is an important aspect of the 5G architecture.} \sphinxAtStartPar \sphinxstyleemphasis{Many of the differences between 5G and Wi\sphinxhyphen{}Fi follow from the differences in spectrum and radio characteristics. For example, cellular deployments, with the expense of spectrum being a given, have historically been carried out by well\sphinxhyphen{}resourced actors who can acquire land, build and connect towers, and hire skilled staff. However, the rise of enterprise 5G and the availability of lightly licensed spectrum is leading to a blurring of the lines between the two approaches.} \end{sphinxShadowBox} \sphinxAtStartPar The rest of this book explains what that means in detail. As an overview, thinking of 5G connectivity as a cloud service means that instead of using purpose\sphinxhyphen{}built devices and telephony\sphinxhyphen{}based operational practices to deliver mobile connectivity, the 5G network is built from commodity hardware, software\sphinxhyphen{}defined networks, and cloud\sphinxhyphen{}based operational practices. And, just as with familiar cloud applications, the end result is a system that increases both feature velocity and operational uniformity. These advantages are available to legacy MNOs, but whether they will fully embrace them is yet to be seen, so we do not limit ourselves to existing stakeholders or business models. In particular, this book focuses on how enterprises can be their own MNOs, or alternatively, acquire private 5G connectivity as a managed cloud service from non\sphinxhyphen{}traditional MNOs. \sphinxAtStartPar To this end, \hyperref[\detokenize{intro:fig-enterprise}]{Figure \ref{\detokenize{intro:fig-enterprise}}} depicts a simplified Private 5G deployment that the rest of this book works toward. At a high level, the figure shows a wide range of enterprise use cases that might take advantage of 5G connectivity, with the data plane of the 5G service running on\sphinxhyphen{}prem (on an edge cloud running within the enterprise), and the control plane of the 5G service running off\sphinxhyphen{}prem (in the global cloud).% \begin{footnote}[1]\sphinxAtStartFootnote We use the term “data plane” in the generic sense in this description. As we’ll see in Chapter 2, the 5G architecture refers to it as “user plane”. % \end{footnote} Enterprise administrators control their service through a management console, much in the same way they might log into an AWS, GCP, or Azure console to control a cloud\sphinxhyphen{}based storage or compute service. Finally, applications are distributed across both edge and centralized clouds, taking advantage of what is commonly referred to as a \sphinxstyleemphasis{hybrid cloud}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide14}.png} \caption{Enterprise\sphinxhyphen{}based deployment of 5G connectivity, running as a managed cloud service.}\label{\detokenize{intro:id4}}\label{\detokenize{intro:fig-enterprise}}\end{figure} \sphinxAtStartPar Hosting a 5G connectivity service on an edge cloud is perfectly aligned with one of the most pronounced trends in cloud computing: moving elements of the cloud from the datacenter to locations that are in close proximity to end users and their devices. Before looking at how to realize 5G on an edge cloud, we start by considering why edge clouds are gaining momentum in the first place. \sphinxAtStartPar The cloud began as a collection of warehouse\sphinxhyphen{}sized datacenters, each of which provided a cost\sphinxhyphen{}effective way to power, cool, and operate a scalable number of servers. Over time, this shared infrastructure lowered the barrier to deploying scalable Internet services, but today, there is increasing pressure to offer low\sphinxhyphen{}latency/high\sphinxhyphen{}bandwidth cloud applications that cannot be effectively implemented in remote datacenters. Augmented Reality (AR), Virtual Reality (VR), Internet of Things (IoT), and Autonomous Vehicles are all examples of this kind of application. Such applications benefit from moving at least part of their functionality out of the datacenter and towards the edge of the network, closer to end users. \sphinxAtStartPar The idea of such deployments is to first collect operational data on assets and infrastructure, from sensors, video feeds and telemetry from machinery. It then applies Machine Learning (ML) or other forms of analysis to this data to gain insights, identify patterns and predict outcomes (e.g., when a device is likely to fail). The final step is to automate industrial processes so as to minimize human intervention and enable remote operations (e.g., power optimization, idling quiescent machinery). The overall goal is to create an IT foundation for continually improving industrial operations through software. \sphinxAtStartPar But precisely where this edge is \sphinxstyleemphasis{physically} located depends on who you ask. If you ask a network operator that already owns and operates thousands of Central Offices, then their Central Offices are an obvious answer. Others might claim the edge is located at the 14,000 Starbucks locations (for example) across the US, and still others might point to the tens of thousands of cell towers spread across the globe. Our approach is to be location agnostic, but to make the discussion concrete, we use enterprises as our exemplar deployment. \sphinxAtStartPar At the same time cloud providers started pursuing edge deployments, network operators began to re\sphinxhyphen{}architect their access network to use the same commodity hardware and best practices in building scalable software as the cloud providers. Such a design, which is sometimes referred to as CORD \sphinxstyleemphasis{(Central Office Re\sphinxhyphen{}architected as a Datacenter)}, supports both the access network and edge services co\sphinxhyphen{}located on a shared cloud platform. This platform is then replicated across hundreds or thousands of operator sites, including Central Offices. \sphinxAtStartPar Traditional network operators did this because they wanted to take advantage of the same economies of scale and feature velocity as cloud providers. CORD gave them a general architecture to work towards, but also an open source Kubernetes\sphinxhyphen{}based reference implementation to model their solutions on. That original implementation of CORD is the direct predecessor to the Aether platform that we use as a reference implementation in this book. \phantomsection\label{\detokenize{intro:reading-cord}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar L. Peterson, \sphinxstyleemphasis{et al}. \sphinxhref{https://wiki.opencord.org/download/attachments/1278027/PETERSON\_CORD.pdf}{Central Office Re\sphinxhyphen{}architected as a Datacenter.}. IEEE Communications, October 2016. \sphinxAtStartPar A.D. Little Report. \sphinxhref{https://www.adlittle.com/en/who-dares-wins}{Who Dares Wins! How Access Transformation Can Fast\sphinxhyphen{}Track Evolution of Operator Production Platforms}. September 2019. \end{sphinxadmonition} \sphinxAtStartPar An important takeaway from this discussion is that to understand how 5G is being implemented, it is helpful to have a working understanding of how clouds are built. This includes the use of \sphinxstyleemphasis{commodity hardware} (both servers and bare\sphinxhyphen{}metal switches), horizontally scalable \sphinxstyleemphasis{microservices} (also referred to as \sphinxstyleemphasis{cloud native}), and \sphinxstyleemphasis{Software\sphinxhyphen{}Defined Networks (SDN)}. It is also helpful to have an appreciation for how cloud software is developed, tested, deployed, and operated, including practices such as \sphinxstyleemphasis{DevOps} and \sphinxstyleemphasis{Continuous Integration / Continuous Deployment (CI/CD)}. We recommend two companion books to help fill the gaps in your understanding of these foundational technologies. \phantomsection\label{\detokenize{intro:reading-devops}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://sdn.systemsapproach.org/}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. November 2021. \sphinxAtStartPar \sphinxhref{https://ops.systemsapproach.org/}{Edge Cloud Operations: A Systems Approach}. June 2022. \end{sphinxadmonition} \section{1.4 Beyond 5G} \label{\detokenize{intro:beyond-5g}} \sphinxAtStartPar From the moment MNOs started rolling out 5G in 2019, people started talking about what comes next. The obvious answer is 6G, but it’s not at all clear that the decadal generations of the past 40 years will continue into the future. Today, you hear alternatives like “NextG” and “Beyond 5G” more often than 6G, which could be a sign that the industry is undergoing a fundamental shift. And there is an argument that we’re in the midst of a sea change that will render the generational distinction largely meaningless. There are two complementary reasons for this, both at the heart of what’s important about Private 5G. \sphinxAtStartPar The first factor is that by adopting cloud technologies, the mobile cellular network is hoping to cash in on the promise of feature velocity. This “agility” story was always included in the early 5G promotional material, as part of the case for why a 5G upgrade would be a worthwhile investment, but the consequence of those technologies now finding their way into the mainstream is that new features can be introduced rapidly and deployed continuously. At some point, the frequency of continual improvements renders generational distinctions irrelevant. \sphinxAtStartPar The second factor is that agility isn’t only about cadence; it’s also about customization. That is, these changes can be introduced bottom\sphinxhyphen{}up—for example by enterprises and their edge cloud partners in the case of Private 5G—without necessarily depending on (or waiting for) a global standardization effort. If an enterprise finds a new use case that requires a specialized deployment, only its Private 5G deployment needs to adopt the necessary changes. Reaching agreement with all the incumbent stakeholders will no longer be a requirement. \sphinxAtStartPar It’s anyone’s guess where this will take us, but it will be interesting to see how this dynamic impacts the role of standardization: what aspects of the mobile network require global agreement and what aspects do not because they can evolve on a case\sphinxhyphen{}by\sphinxhyphen{}case basis. While standards often spur innovation (TCP and HTTP are two great examples from the Internet experience), sometimes standards actually serve as a barrier to competition, and hence, innovation. Now that software is eating the mobile cellular network—with Private 5G deployed in enterprises likely setting the pace—we will learn which standards are which. \sphinxAtStartPar In summary, that 5G is on an evolutionary path is the central theme of this book. We call attention to its importance here, and revisit the topic throughout the book. We are writing this book for \sphinxstyleemphasis{system generalists}, with the goal of helping bring a community that understands a broad range of systems issues (but knows little or nothing about the cellular network) up to speed so they can play a role in its evolution. This is a community that understands both feature velocity and best practices in building robust scalable systems, and so has an important role to play in bringing all of 5G’s potential to fruition. \chapter{Chapter 2: Architecture} \label{\detokenize{arch:chapter-2-architecture}}\label{\detokenize{arch::doc}} \sphinxAtStartPar This chapter identifies the main architectural components of the mobile cellular network. We need to introduce some terminology to do this, which can be confusing for those whose networking background comes from the Internet. This is partly because some of what needs to happen in a mobile network, such as keeping track of which base station is serving a given mobile device, doesn’t really have a parallel in fixed networks. On top of that, the terminology came out of the 3GPP standardization process, which was historically concerned with telephony and almost completely disconnected from the IETF and other Internet\sphinxhyphen{}related efforts. To further confuse matters, 3GPP terminology often changes with each generation (e.g., a base station is called eNB in 4G and gNB in 5G). We address situations like this by using generic terminology (e.g., base station), and referencing the 3GPP\sphinxhyphen{}specific counterpart only when the distinction is helpful. This example is only the tip of the terminology iceberg. Marcin Dryjanski’s blog post gives a broader perspective on the complexity of terminology in 5G. \phantomsection\label{\detokenize{arch:reading-terminology}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar Marcin Dryjanski. \sphinxhref{https://www.grandmetric.com/blog/2018/07/14/lte-and-5g-differences-system-complexity/}{LTE and 5G Differences: System Complexity}. July 2018. \end{sphinxadmonition} \section{2.1 Overview} \label{\detokenize{arch:overview}} \sphinxAtStartPar The mobile cellular network provides wireless connectivity to devices that are (potentially) on the move. These devices, which are known as \sphinxstyleemphasis{User Equipment (UE)}, have traditionally corresponded to mobile phones and tablets, but increasingly include cars, drones, industrial and agricultural machines, robots, home appliances, medical devices, and so on. In some cases, the UEs may be devices that do not move, e.g., router interfaces using cellular connectivity to provide broadband access to remote dwellings. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide2}.png} \caption{Mobile cellular networks consist of a Radio Access Network (RAN) and a Mobile Core.}\label{\detokenize{arch:id3}}\label{\detokenize{arch:fig-cellular}}\end{figure} \sphinxAtStartPar As shown in \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}, the mobile cellular network consists of two main subsystems: the \sphinxstyleemphasis{Radio Access Network (RAN)} and the \sphinxstyleemphasis{Mobile Core}. The RAN manages the radio resources (i.e., spectrum), making sure it is used efficiently and meets the quality of service (QoS) requirements of every user. It corresponds to a distributed collection of base stations. As noted above, these are cryptically named \sphinxstyleemphasis{eNodeB} or \sphinxstyleemphasis{eNB} (which is short for \sphinxstyleemphasis{evolved Node B}) in 4G. In 5G, base stations are known as \sphinxstyleemphasis{gNB}, where the “g” stands for \sphinxstyleemphasis{next Generation}. \sphinxAtStartPar The Mobile Core is a bundle of functionality (conventionally packaged as one or more devices) that serves several purposes. \begin{itemize} \item {} \sphinxAtStartPar Authenticates devices prior to attaching them to the network \item {} \sphinxAtStartPar Provides Internet (IP) connectivity for both data and voice services. \item {} \sphinxAtStartPar Ensures this connectivity fulfills the promised QoS requirements. \item {} \sphinxAtStartPar Tracks user mobility to ensure uninterrupted service. \item {} \sphinxAtStartPar Tracks subscriber usage for billing and charging. \end{itemize} \sphinxAtStartPar For readers familiar with the Internet architecture and Wi\sphinxhyphen{}Fi as a common access technology, some of these functions might look a bit surprising. For example, Wi\sphinxhyphen{}Fi, like most of the Internet, normally provides a best\sphinxhyphen{}effort service, whereas cellular networks often aim to deliver some sort of QoS guarantee. Tracking subscribers for both mobility and billing are also not the sort of things we tend to think about in the Internet, but they are considered important functions for cellular networks. The reasons for these differences are numerous, including the typically large costs of acquiring cellular spectrum and maintaining the infrastructure to use it such as radio towers. With that large investment, there is a desire to recoup costs by charging subscribers, which in turn leads to making some sort of service guarantees to those subscribers to justify the cost. There is also a need to maximize the efficiency of spectrum usage. Much of the complexity of the mobile core follows from these requirements being imposed by service providers. Even when we get to enterprises running their own 5G networks, they still need to manage the usage of spectrum to obtain the benefits of 5G over Wi\sphinxhyphen{}Fi, such as more predictable control over latency and bandwidth. \sphinxAtStartPar Note that Mobile Core is another example of a generic term. In 4G it was called the \sphinxstyleemphasis{Evolved Packet Core (EPC)} and in 5G it is called the \sphinxstyleemphasis{5G Core (5GC)}. Moreover, even though the word “Core” is in its name, the Mobile Core runs near the edge of the network, effectively providing a bridge between the RAN in some geographic area and the greater IP\sphinxhyphen{}based Internet. 3GPP provides significant flexibility in how the Mobile Core is geographically deployed, ranging from minimal deployments (the RAN and the mobile core can be co\sphinxhyphen{}located) to areas that are hundreds of kilometers wide. A common model is that an instantiation of the Mobile Core serves a metropolitan area. The corresponding RAN would then span several dozens (or even hundreds) of cell towers in that geographic area. \sphinxAtStartPar Taking a closer look at \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}, we see that a \sphinxstyleemphasis{Backhaul Network} interconnects the base stations that implement the RAN with the Mobile Core. This network is typically wired, may or may not have the ring topology shown in the figure, and is often constructed from commodity components found elsewhere in the Internet. For example, the \sphinxstyleemphasis{Passive Optical Network (PON)} that implements Fiber\sphinxhyphen{}to\sphinxhyphen{}the\sphinxhyphen{}Home is a prime candidate for implementing the RAN backhaul, with the RAN effectively running as an \sphinxstyleemphasis{overlay} on top of whatever technology is used. Switched ethernet, such as you might find in an enterprise, is another suitable choice. The backhaul network is obviously a necessary part of the RAN, but it is an implementation choice and not prescribed by the 3GPP standard. \sphinxAtStartPar Although 3GPP specifies all the elements that implement the RAN and Mobile Core in an open standard—including sub\sphinxhyphen{}layers we have not yet introduced—network operators have historically bought proprietary implementations of each subsystem from a single vendor. This lack of an open source implementation contributes to the perceived “opaqueness” of the mobile cellular network in general, and the RAN in particular. And while it is true that base stations contain sophisticated algorithms for scheduling transmission on the radio spectrum—algorithms that are considered valuable intellectual property of the equipment vendors—there is significant opportunity to open and disaggregate both the RAN and the Mobile Core. This book gives a recipe for how to do exactly that. \sphinxAtStartPar Before getting to those details, we have three more architectural concepts to introduce. First, \hyperref[\detokenize{arch:fig-cups}]{Figure \ref{\detokenize{arch:fig-cups}}} redraws components from \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}} to highlight the fact that a base station has an analog component (depicted by an antenna) and a digital component (depicted by a processor pair). This book mostly focuses on the latter, but we introduce enough information about the over\sphinxhyphen{}the\sphinxhyphen{}air radio transmission to appreciate its impact on the overall architecture. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide3}.png} \caption{Mobile Core divided into a Control Plane and a User Plane, an architectural feature known as CUPS: Control and User Plane Separation.}\label{\detokenize{arch:id4}}\label{\detokenize{arch:fig-cups}}\end{figure} \sphinxAtStartPar The second concept, also depicted in \hyperref[\detokenize{arch:fig-cups}]{Figure \ref{\detokenize{arch:fig-cups}}}, is to partition the Mobile Core into a \sphinxstyleemphasis{Control Plane} and \sphinxstyleemphasis{User Plane}. This is similar to the control/data plane split that anyone familiar with the Internet would recognize, and draws in particular on the ideas of software\sphinxhyphen{}defined networking (SDN) by placing control and user planes in separate devices. 3GPP has introduced a corresponding acronym—\sphinxstyleemphasis{CUPS, Control and User Plane Separation}—to denote this idea. One motivation for CUPS is to enable control plane resources and data plane resources to be scaled independently of each other. \sphinxAtStartPar Finally, one of the key aspirational goals of 5G is the ability to segregate traffic for different usage domains into isolated \sphinxstyleemphasis{network slices}, each of which delivers a different level of service to a collection of devices and applications. Thinking of a network slice as a wireless version of a virtual network is a fair approximation, although as we’ll see in later chapters, the implementation details differ. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide4}.png} \caption{Different usage domains (e.g., IoT and Video Streaming) instantiate distinct \sphinxstyleemphasis{network slices} to connect a set of devices with one or more applications.}\label{\detokenize{arch:id5}}\label{\detokenize{arch:fig-slice}}\end{figure} \sphinxAtStartPar For example, \hyperref[\detokenize{arch:fig-slice}]{Figure \ref{\detokenize{arch:fig-slice}}} shows two slices, one supporting IoT workloads and the other supporting multimedia streaming traffic. As we’ll see throughout the book, an important question is how slicing is realized end\sphinxhyphen{}to\sphinxhyphen{}end, across the radio, the RAN, and the Mobile Core. This is done through a combination of allocating distinct resources to each slice and scheduling shared resources on behalf of a set of slices. \section{2.2 Radio Transmission} \label{\detokenize{arch:radio-transmission}} \sphinxAtStartPar Before describing the RAN and Mobile Core subsystems, we first call attention to the obvious: that the base stations that comprise the RAN communicate with UEs via electromagnetic radio waves. This book is not about the physics of this over\sphinxhyphen{}the\sphinxhyphen{}air communication, and only skims the surface of the information theory that underlies it. But identifying the abstract properties of wireless communication is an essential foundation for understanding the rest of the 5G architecture. \sphinxAtStartPar If you imagine the base stations as implementing a multi\sphinxhyphen{}layer protocol stack (which, as we’ll see in Chapter 4, they do), then radio transmission is the responsibility of the bottom\sphinxhyphen{}most layers of that stack, where (a) digital/analog conversion happens, and (b) analog radio waves are transmitted/received. Chapter 3 introduces radio transmission with enough specificity to lay the necessary foundation, so we’re able to understand all the layers that come above it. \sphinxAtStartPar For the purposes of this chapter, we only need to understand the following. First, the RAN is responsible for managing how the radio spectrum is shared among thousands of UEs connected to hundreds of base stations in a geographic region. The primary purpose of Chapter 3 is to establish an abstract interface by which the RAN can manage that spectrum without having to worry about the details of waveforms, modulation, or coding algorithms. All important topics, to be sure, but in the realm of information theory rather than system design that is the focus of this book. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=300\sphinxpxdimen]{{Slide5}.png} \caption{Abstractly, measures of signal quality (CQI) and declarations of intended data delivery quality (5QI) are passed up and down the RAN stack.}\label{\detokenize{arch:id6}}\label{\detokenize{arch:fig-quality}}\end{figure} \sphinxAtStartPar Second, there are two important pieces of information shared between the higher layers of the base station protocol stack that manages the RAN as a whole, and the lower layers of the stack that manage radio transmissions on a particular base station. One is the signal\sphinxhyphen{}to\sphinxhyphen{}noise ratio that the base station observes when communicating with each UE. This is called the \sphinxstyleemphasis{Channel Quality Indicator (CQI)} and it is passed \sphinxstyleemphasis{up} from the radio. The other is the quality of service the network wants to give a particular UE. This is called the \sphinxstyleemphasis{5G QoS Identifier (5QI)} and it is passed \sphinxstyleemphasis{down} to the radio. This abstract summary, as shown in \hyperref[\detokenize{arch:fig-quality}]{Figure \ref{\detokenize{arch:fig-quality}}}, is sufficient to introduce the RAN and Mobile Core. We will fill in more details about both of these parameters in Chapter 3. \begin{sphinxShadowBox} \sphinxstylesidebartitle{Uniqueness of Wireless Links} \sphinxAtStartPar \sphinxstyleemphasis{While it is common in networking to abstract the link layer by treating the link as something that just delivers packets at some rate from point A to point B, there are important differences between wireless links and wired links that cannot be entirely abstracted away at higher layers. This is especially true when mobile devices are involved, as the quality of a link will vary depending on the distance between transmitter and receiver, the relative velocity of the endpoints, reflections of radio waves from other objects, and interference from other transmitters. All of these factors come into play in determining the Channel Quality Indicator (CQI).} \sphinxAtStartPar \sphinxstyleemphasis{Further complicating the picture in a mobile network is that a given UE is often within reach of more than one base station, presenting the option to handover the UE from one base station to another. The decision to do so is not just a matter of picking the base station with the best channel quality, but rather a matter of trying to optimize the whole system, in which the goal is to support as many UEs as possible at the desired quality level given the available spectrum and coverage.} \end{sphinxShadowBox} \sphinxAtStartPar Finally, like the rest of the mobile cellular network, the radio comes with a set of acronyms, with \sphinxstyleemphasis{LTE (Long\sphinxhyphen{}Term Evolution)} and \sphinxstyleemphasis{NR (New Radio)} being the two most widely known. These are marketing terms commonly associated with the radio technology for 4G and 5G, respectively. They are important only in the sense that many of the new features promised by 5G can be directly attributed to improvements in the underlying radio technology. For our purposes, the key is the set of new \sphinxstyleemphasis{use cases} the upgraded radio technology enables, and why. We introduce these improvements to the radio in Chapter 3, and tie them to the use cases they enable. Subsequent chapters will then explain how the RAN and Mobile Core need to evolve so as to deliver on this potential. \section{2.3 Radio Access Network} \label{\detokenize{arch:radio-access-network}} \sphinxAtStartPar We now describe the RAN by sketching the role each base station plays. Keep in mind this is like describing the Internet by explaining how a router works—not an unreasonable place to start, but it doesn’t fully do justice to the end\sphinxhyphen{}to\sphinxhyphen{}end story. \sphinxAtStartPar First, each base station establishes the wireless channel for a subscriber’s UE upon power\sphinxhyphen{}up or upon handover when the UE is active. This channel is released when the UE remains idle for a predetermined period of time. Using 3GPP terminology, this wireless channel is said to provide a \sphinxstyleemphasis{radio bearer}. The term “bearer” has historically been used in telecommunications (including early wireline technologies such as ISDN) to denote a data channel, as opposed to a channel that carries signaling information. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide6}.png} \caption{UE detects (and connects to) base station.}\label{\detokenize{arch:id7}}\label{\detokenize{arch:fig-active-ue}}\end{figure} \sphinxAtStartPar Second, each base station establishes “3GPP Control Plane” connectivity between the UE and the corresponding Mobile Core Control Plane component, and forwards signaling traffic between the two. This signaling traffic enables UE authentication, registration, and mobility tracking. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide7}.png} \caption{Base Station establishes control plane connectivity between each UE and the Mobile Core.}\label{\detokenize{arch:id8}}\label{\detokenize{arch:fig-control-plane}}\end{figure} \sphinxAtStartPar Third, for each active UE, the base station establishes one or more tunnels to the corresponding Mobile Core User Plane component. \hyperref[\detokenize{arch:fig-user-plane}]{Figure \ref{\detokenize{arch:fig-user-plane}}} shows just two (one for voice and one for data), and while in practice 4G was limited to just two, 5G aspires to support many such tunnels as part of a generalized network slicing mechanism. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide8}.png} \caption{Base station establishes one or more tunnels between each UE and the Mobile Core’s User Plane (known in 3GPP terms as PDU session).}\label{\detokenize{arch:id9}}\label{\detokenize{arch:fig-user-plane}}\end{figure} \sphinxAtStartPar Fourth, the base station forwards both control and user plane packets between the Mobile Core and the UE. These packets are tunneled over SCTP/IP and GTP/UDP/IP, respectively. SCTP (Stream Control Transport Protocol) is an alternative reliable transport to TCP, tailored to carry signaling (control) information for telephony services. GTP (a nested acronym corresponding to (General Packet Radio Service) Tunneling Protocol) is a 3GPP\sphinxhyphen{}specific tunneling protocol designed to run over UDP. \sphinxAtStartPar It is noteworthy that connectivity between the RAN and the Mobile Core is IP\sphinxhyphen{}based. This was introduced as one of the main changes between 3G and 4G. Prior to 4G, the internals of the cellular network were circuit\sphinxhyphen{}based, which is not surprising given its origins as a voice network. This also helps to explain why in Section 2.1 we characterized the RAN Backhaul as an overlay running on top of some Layer 2 technology. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide9}.png} \caption{Base Station to Mobile Core (and Base Station to Base Station) control plane tunneled over SCTP/IP and user plane tunneled over GTP/UDP/IP.}\label{\detokenize{arch:id10}}\label{\detokenize{arch:fig-tunnels}}\end{figure} \sphinxAtStartPar Fifth, each base station coordinates UE handovers with neighboring base stations, using direct station\sphinxhyphen{}to\sphinxhyphen{}station links. Exactly like the station\sphinxhyphen{}to\sphinxhyphen{}core connectivity shown in the previous figure, these links are used to transfer both control plane (SCTP over IP) and user plane (GTP over UDP/IP) packets. The decision as to when to do a handover is based on the CQI values being reported by the radio on each of the base stations within range of the UE, coupled with the 5QI value those base stations know the RAN has promised to deliver to the UE. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide10}.png} \caption{Base Stations cooperate to implement UE hand over.}\label{\detokenize{arch:id11}}\label{\detokenize{arch:fig-handover}}\end{figure} \sphinxAtStartPar Sixth, the base stations coordinate wireless multi\sphinxhyphen{}point transmission to a UE from multiple base stations, which may or may not be part of a UE handover from one base station to another. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide11}.png} \caption{Base Stations cooperate to implement multipath transmission (link aggregation) to UEs.}\label{\detokenize{arch:id12}}\label{\detokenize{arch:fig-link-aggregation}}\end{figure} \sphinxAtStartPar The main takeaway is that the base station can be viewed as a specialized forwarder. In the Internet\sphinxhyphen{}to\sphinxhyphen{}UE direction, it fragments outgoing IP packets into physical layer segments and schedules them for transmission over the available radio spectrum, and in the UE\sphinxhyphen{}to\sphinxhyphen{}Internet direction it assembles physical layer segments into IP packets and forwards them (over a GTP/UDP/IP tunnel) to the upstream user plane of the Mobile Core. Also, based on observations of the wireless channel quality and per\sphinxhyphen{}subscriber policies, it decides whether to (a) forward outgoing packets directly to the UE, (b) indirectly forward packets to the UE via a neighboring base station, or (c) utilize multiple paths to reach the UE. The third case has the option of either spreading the physical payloads across multiple base stations or across multiple carrier frequencies of a single base station (including Wi\sphinxhyphen{}Fi). \sphinxAtStartPar In other words, the RAN as a whole (i.e., not just a single base station) not only supports handovers (an obvious requirement for mobility), but also \sphinxstyleemphasis{link aggregation} and \sphinxstyleemphasis{load balancing}, mechanisms that are similar to those found in other types of networks. These functions imply a global decision\sphinxhyphen{}making process, whereby it’s possible to forward traffic to a different base station (or to multiple base stations) in an effort to make efficient use of the radio spectrum over a larger geographic area. We will revisit how such RAN\sphinxhyphen{}wide (global) decisions can be made using SDN techniques in Chapter 4. \section{2.4 Mobile Core} \label{\detokenize{arch:mobile-core}} \sphinxAtStartPar At the most basic level, the function of the Mobile Core is to provide packet data network connectivity to mobile subscribers, i.e., connect them to the Internet. (The mobile network assumes that multiple packet data networks can exist, but in practice the Internet is the one that matters). As we noted above, there is more to providing this connectivity than meets the eye: the Mobile Core ensures that subscribers are authenticated and aims to deliver the service qualities to which they have subscribed. As subscribers may move around among base station coverage areas, the Mobile Core needs to keep track of their whereabouts at the granularity of the serving base station. The reasons for this tracking are discussed further in Chapter 5. It is this support for security, mobility, and QoS that differentiates the cellular network from Wi\sphinxhyphen{}Fi. \sphinxAtStartPar We start with the security architecture, which is grounded in two trust assumptions. First, each base station trusts that it is connected to the Mobile Core by a secure private network, over which it establishes the tunnels introduced in \hyperref[\detokenize{arch:fig-tunnels}]{Figure \ref{\detokenize{arch:fig-tunnels}}}: a GTP/UDP/IP tunnel to the Core’s User Plane (Core\sphinxhyphen{}UP) and a SCTP/IP tunnel to the Core’s Control Plane (Core\sphinxhyphen{}CP). Second, each UE has an operator\sphinxhyphen{}provided SIM card, which contains information that uniquely identifies the subscriber and includes a secret key that the UE uses to authenticate itself. \sphinxAtStartPar The identifier burned into each SIM card, called an \sphinxstyleemphasis{IMSI (International Mobile Subscriber Identity)}, is a globally unique identifier for every device connected to the global mobile network. Each IMSI is a 64\sphinxhyphen{}bit, self\sphinxhyphen{}describing identifier, which is to say, it includes a \sphinxstyleemphasis{Format} field that effectively serves as a mask for extracting other relevant fields. For example, the following is the interpretation we assume in this book (where IMSIs are commonly represented as an up to 15\sphinxhyphen{}digit decimal number): \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{MCC:} Mobile Country Code (3\sphinxhyphen{}digit decimal number). \item {} \sphinxAtStartPar \sphinxstylestrong{MNC:} Mobile Network Code (2 or 3\sphinxhyphen{}digit decimal number). \item {} \sphinxAtStartPar \sphinxstylestrong{ENT:} Enterprise Code (3\sphinxhyphen{}digit decimal number). \item {} \sphinxAtStartPar \sphinxstylestrong{SUB:} Subscriber (6\sphinxhyphen{}digit decimal number). \end{itemize} \sphinxAtStartPar The first two fields (\sphinxstyleemphasis{MCC}, \sphinxstyleemphasis{MNC}) are universally understood to uniquely identify the MNO, while that last two fields are one example of how an MNO might use additional hierarchical structure to uniquely identify every device it serves. We are working towards delivering 5G connectivity to enterprises (hence the \sphinxstyleemphasis{ENT} field), but other MNOs might assign the last 9 digits using some other structure. \sphinxAtStartPar The \sphinxstyleemphasis{MCC/MNC} pair—which is also called the \sphinxstyleemphasis{Public Land Mobile Network (PLMN)} identifier—plays a role in roaming: when a UE tries to connect to a “foreign network” those fields are used to find the “home network”, where the rest of the IMSI leads to a subscriber profile that says whether or not roaming is enabled for this device. The following walks through what happens when a device connects to its home network; more information about the global ramifications is given at the end of the section. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide12}.png} \caption{Sequence of steps to establish secure Control and User Plane channels.}\label{\detokenize{arch:id13}}\label{\detokenize{arch:fig-secure}}\end{figure} \sphinxAtStartPar \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} shows the per\sphinxhyphen{}UE connection sequence. When a UE first becomes active, it communicates with a nearby base station over a temporary (unauthenticated) radio link (Step 1). The base station forwards the request to the Core\sphinxhyphen{}CP over the existing SCTP connection, and the Core\sphinxhyphen{}CP (assuming it recognizes the IMSI) initiates an authentication protocol with the UE (Step 2). 3GPP identifies a set of options for authentication and encryption, where the actual protocols used are an implementation choice. For example, \sphinxstyleemphasis{Advanced Encryption Standard} (AES) is one of the options for encryption. Note that this authentication exchange is initially in the clear since the base station to UE link is not yet secure. \sphinxAtStartPar Once the UE and Core\sphinxhyphen{}CP are satisfied with each other’s identity, the Core\sphinxhyphen{}CP informs the other 5GC components of the parameters they will need to service the UE (Step 3). This includes: (a) instructing the Core\sphinxhyphen{}UP to initialize the user plane (e.g., assign an IP address to the UE and set the appropriate 5QI); (b) instructing the base station to establish an encrypted channel to the UE; and (c) giving the UE the symmetric key it will need to use the encrypted channel with the base station. The symmetric key is encrypted using the public key of the UE (so only the UE can decrypt it, using its secret key). Once complete, the UE can use the end\sphinxhyphen{}to\sphinxhyphen{}end user plane channel through the Core\sphinxhyphen{}UP (Step 4). \sphinxAtStartPar There are three additional details of note about this process. First, the secure control channel between the UE and the Core\sphinxhyphen{}CP set up during Step 2 remains available, and is used by the Core\sphinxhyphen{}CP to send additional control instructions to the UE during the course of the session. In other words, unlike the Internet, the network is able to “control” the communication settings in edge devices. \sphinxAtStartPar Second, the user plane channel established during Step 4 is referred to as the \sphinxstyleemphasis{Default Data Radio Bearer}, but additional channels can be established between the UE and Core\sphinxhyphen{}UP, each with a potentially different 5QI. This might be done on an application\sphinxhyphen{}by\sphinxhyphen{}application basis, for example, based on policies present in the Core\sphinxhyphen{}CP for packets that require special/different treatment. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide13}.png} \caption{Sequence of per\sphinxhyphen{}hop tunnels involved in an end\sphinxhyphen{}to\sphinxhyphen{}end User Plane channel.}\label{\detokenize{arch:id14}}\label{\detokenize{arch:fig-per-hop}}\end{figure} \sphinxAtStartPar In practice, these per\sphinxhyphen{}flow tunnels are often bundled into a single inter\sphinxhyphen{}component tunnel, which makes it impossible to differentiate the level of service given to any particular end\sphinxhyphen{}to\sphinxhyphen{}end UE channel. This is a limitation of 4G that 5G has ambitions to correct as part of its support for network slicing. \sphinxAtStartPar Support for mobility can now be understood as the process of re\sphinxhyphen{}executing one or more of the steps shown in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} as the UE moves throughout the RAN. The unauthenticated link indicated by (1) allows the UE to be known to all base stations within range. (We refer to these as \sphinxstyleemphasis{potential links} in later chapters.) Based on the signal’s measured CQI, the base stations communicate directly with each other to make a handover decision. Once made, the decision is then communicated to the Mobile Core, re\sphinxhyphen{}triggering the setup functions indicated by (3), which in turn re\sphinxhyphen{}builds the user plane tunnel between the base station and the Core\sphinxhyphen{}UP shown in \hyperref[\detokenize{arch:fig-per-hop}]{Figure \ref{\detokenize{arch:fig-per-hop}}}. One of the most unique features of the cellular network is that the Mobile Core’s user plane buffers data while idle UEs are transiting to active state, thereby avoiding dropped packets and subsequent end\sphinxhyphen{}to\sphinxhyphen{}end retransmissions. \sphinxAtStartPar In other words, the mobile cellular network maintains the \sphinxstyleemphasis{UE session} in the face of mobility (corresponding to the control and data channels depicted by (2) and (4) in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}}, respectively), but it is able to do so only when the same Mobile Core serves the UE (i.e., only the base station changes). This would typically be the case for a UE moving within a metropolitan area. Moving between metro areas—and hence, between Mobile Cores—is indistinguishable from power cycling a UE. The UE is assigned a new IP address and no attempt is made to buffer and subsequently deliver in\sphinxhyphen{}flight data. Independent of mobility, but relevant to this discussion, any UE that becomes inactive for a period of time also loses its session, with a new session established and a new IP address assigned when the UE becomes active again. \sphinxAtStartPar Note that this session\sphinxhyphen{}based approach can be traced to the mobile cellular network’s roots as a connection\sphinxhyphen{}oriented network. An interesting thought experiment is whether the Mobile Core will continue to evolve so as to better match the connectionless assumptions of the Internet protocols that typically run on top of it. \sphinxAtStartPar We conclude this overview of the Mobile Core by returning to the role it plays in implementing a \sphinxstyleemphasis{global} mobile network. It is probably already clear that each MNO implements a database of subscriber information, allowing it to map an IMSI to a profile that records what services (roaming, data plane, hot spot support) the subscriber is paying for. This record also includes the international phone number for the device. How this database is realized is an implementation choice (of which we’ll see an example in Chapter 5), but 3GPP defines an interface by which one Mobile Core (running on behalf of one MNO) queries another Mobile Core (running on behalf of some other MNO), to map between the IMSI, the phone number, and the subscriber profile. \section{2.5 Managed Cloud Service} \label{\detokenize{arch:managed-cloud-service}} \sphinxAtStartPar The architectural overview presented up to this point focuses on the functional elements of the mobile cellular network. We now turn our attention to how this functionality is operationalized, and we do so in a decidedly software\sphinxhyphen{}defined and cloud\sphinxhyphen{}native way. This lays the foundation for the rest of the book, which builds towards supporting 5G connectivity as a managed cloud service. This is a marked change from the conventional Telco approach, whereby an operator bought purpose\sphinxhyphen{}built devices from a handful of vendors, and then managed them using the legacy OSS/BSS machinery that was originally designed for the telephony network.% \begin{footnote}[1]\sphinxAtStartFootnote OSS/BSS stands for Operation Support System / Business Support System, and even traditional MNOs are now re\sphinxhyphen{}imagining them by adopting cloud practices. But this transition is a slow process due to all the legacy systems the Telcos need to continue supporting. % \end{footnote} \sphinxAtStartPar When we talk about “operationalizing” a network, we are referring to a substantial system that operators (whether they are traditional MNOs or cloud service providers) use to activate and manage all the constituent components (whether they are purpose\sphinxhyphen{}built devices or software running on commodity hardware). And because these network operators are people, one high\sphinxhyphen{}level summary is that this management layer (whether it is an OSS/BSS or a cloud orchestrator) provides a way to map high\sphinxhyphen{}level \sphinxstyleemphasis{Intents} onto low\sphinxhyphen{}level \sphinxstyleemphasis{Actions}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=300\sphinxpxdimen]{{Slide21}.png} \caption{High\sphinxhyphen{}level summary of the role operationalization plays in a network deployment.}\label{\detokenize{arch:id15}}\label{\detokenize{arch:fig-intent}}\end{figure} \sphinxAtStartPar This overview, as illustrated in \hyperref[\detokenize{arch:fig-intent}]{Figure \ref{\detokenize{arch:fig-intent}}}, is obviously quite abstract. To make the discussion more concrete, we use an open source implementation, called Aether, as an example. Aether is a Kubernetes\sphinxhyphen{}based edge cloud, augmented with a 5G\sphinxhyphen{}based connectivity service. Aether is targeted at enterprises that want to take advantage of 5G connectivity in support of edge applications that require predictable, low\sphinxhyphen{}latency connectivity. In short, “Kubernetes\sphinxhyphen{}based” means Aether is able to host container\sphinxhyphen{}based services, with Kubernetes being the platform used to orchestrate the services, and “5G\sphinxhyphen{}based connectivity” means Aether is able to connect those services to mobile devices throughout the enterprise’s physical environment. \sphinxAtStartPar Aether supports this combination by implementing both the RAN and the user plane of the Mobile Core on\sphinxhyphen{}prem, as cloud\sphinxhyphen{}native workloads co\sphinxhyphen{}located on the Aether cluster. This is often referred to as \sphinxstyleemphasis{local breakout} because it enables direct communication between mobile devices and edge applications without data traffic leaving the enterprise, in contrast to what would happen with standard, operator\sphinxhyphen{}provided 5G service. This scenario is depicted in \hyperref[\detokenize{arch:fig-hybrid}]{Figure \ref{\detokenize{arch:fig-hybrid}}}, which shows unnamed edge applications running on\sphinxhyphen{}prem. Those edge applications might include the local processing of sensor data or control applications for the IoT devices, for example. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide31}.png} \caption{Overview of Aether as a hybrid cloud, with edge apps and the 5G data plane (called \sphinxstyleemphasis{local breakout}) running on\sphinxhyphen{}prem and various management and control\sphinxhyphen{}related workloads running in a central cloud.}\label{\detokenize{arch:id16}}\label{\detokenize{arch:fig-hybrid}}\end{figure} \sphinxAtStartPar The approach includes both edge (on\sphinxhyphen{}prem) and centralized (off\sphinxhyphen{}prem) components. This is true for edge apps, which often have a centralized counterpart running in a commodity cloud. It is also true for the 5G Mobile Core, where the on\sphinxhyphen{}prem User Plane (UP) is paired with a centralized Control Plane (CP). The central cloud shown in this figure might be private (i.e., operated by the enterprise), public (i.e., operated by a commercial cloud provider), or some combination of the two (i.e., not all centralized elements need to run in the same cloud). \sphinxAtStartPar Also shown in \hyperref[\detokenize{arch:fig-hybrid}]{Figure \ref{\detokenize{arch:fig-hybrid}}} is a centralized \sphinxstyleemphasis{Control and Management Platform}. This is Aether’s version of the “Management Layer” depicted in \hyperref[\detokenize{arch:fig-intent}]{Figure \ref{\detokenize{arch:fig-intent}}}, and it represents all the functionality needed to offer Aether as a managed cloud service, with system administrators using a portal exported by this platform to operate the underlying infrastructure and services within their enterprise. \sphinxAtStartPar Once we deconstruct the individual components in more details in the next three chapters, we return to the question of how the resulting set of components can be assembled into an operational edge cloud in Chapter 6. The end result is 5G connectivity as a managed cloud service. \chapter{Chapter 3: Radio Transmission} \label{\detokenize{radio:chapter-3-radio-transmission}}\label{\detokenize{radio::doc}} \sphinxAtStartPar For anyone familiar with wireless access technologies that offer a best\sphinxhyphen{}effort service, such as Wi\sphinxhyphen{}Fi, the cellular network presents a notable contrast. This is mostly because cellular networks carefully allocate available radio spectrum among many users (or more precisely, UEs), aiming to deliver a certain quality to all active users in a coverage area, while also allowing those users to remain connected while moving. They also aim to maximize the efficiency of spectrum usage, as it is a finite and often costly resource. This has resulted in a highly dynamic and adaptive approach, in which coding, modulation and scheduling play a central role. 5G takes the cellular approach to managing radio resources to a new level of sophistication. \sphinxAtStartPar Wireless transmission is full of challenges that don’t arise in wired networks. Interference can arise from many sources ranging from microwave ovens to baby monitors, while radio signals reflect off objects such as buildings and vehicles. Some objects absorb wireless signals. The properties of the wireless channel vary over time, and depending on the frequency in use. Much of the design of cellular radio systems is motivated by the need to deal with these challenges. \sphinxAtStartPar As we will see in this chapter, mobile cellular networks use a reservation\sphinxhyphen{}based strategy, whereas Wi\sphinxhyphen{}Fi is contention\sphinxhyphen{}based. This difference is rooted in each system’s fundamental assumption about utilization: Wi\sphinxhyphen{}Fi assumes a lightly loaded network (and hence optimistically transmits when the wireless link is idle and backs off if contention is detected), while 5G assumes (and strives for) high utilization (and hence explicitly assign different users to different “shares” of the available radio spectrum). \sphinxAtStartPar The fact that 5G controls spectrum allocation carefully is central to its ability to deliver guarantees of bandwidth and latency more predictably than Wi\sphinxhyphen{}Fi. This in turn is why there is so much interest in using 5G for mission\sphinxhyphen{}critical applications. \sphinxAtStartPar We start by giving a short primer on radio transmission as a way of laying a foundation for understanding the rest of the 5G architecture. The following is not a substitute for a theoretical treatment of the topic, but is instead intended as a way of grounding the systems\sphinxhyphen{}oriented description of 5G that follows in the reality of wireless communication. \section{3.1 Coding and Modulation} \label{\detokenize{radio:coding-and-modulation}} \sphinxAtStartPar The mobile channel over which digital data needs to be reliably transmitted brings a number of impairments, including noise, attenuation, distortion, fading, and interference. This challenge is addressed by a combination of coding and modulation, as depicted in \hyperref[\detokenize{radio:fig-modulation}]{Figure \ref{\detokenize{radio:fig-modulation}}}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide141}.png} \caption{The role of coding and modulation in mobile communication.}\label{\detokenize{radio:id5}}\label{\detokenize{radio:fig-modulation}}\end{figure} \sphinxAtStartPar At its core, coding inserts extra bits into the data to help recover from all the environmental factors that interfere with signal propagation. This typically implies some form of \sphinxstyleemphasis{Forward Error Correction} (e.g., turbo codes, polar codes). Modulation then generates signals that represent the encoded data stream, and it does so in a way that matches the channel characteristics: It first uses a digital modulation signal format that maximizes the number of reliably transmitted bits every second based on the specifics of the observed channel impairments; it next matches the transmission bandwidth to channel bandwidth using pulse shaping; and finally, it uses RF modulation to transmit the signal as an electromagnetic wave over an assigned \sphinxstyleemphasis{carrier frequency.} \sphinxAtStartPar For a deeper appreciation of the challenges of reliably transmitting data by propagating radio signals through the air, consider the scenario depicted in \hyperref[\detokenize{radio:fig-multipath}]{Figure \ref{\detokenize{radio:fig-multipath}}}, where the signal bounces off various stationary and moving objects, following multiple paths from the transmitter to the receiver, who may also be moving. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide15}.png} \caption{Signals propagate along multiple paths from transmitter to receiver.}\label{\detokenize{radio:id6}}\label{\detokenize{radio:fig-multipath}}\end{figure} \sphinxAtStartPar As a consequence of these multiple paths, the original signal arrives at the receiver spread over time, as illustrated in \hyperref[\detokenize{radio:fig-coherence}]{Figure \ref{\detokenize{radio:fig-coherence}}}. Empirical evidence shows that the Multipath Spread—the time between the first and last signals of one transmission arriving at the receiver—is 1\sphinxhyphen{}10μs in urban environments and 10\sphinxhyphen{}30μs in suburban environments. These multipath signals can interfere with each other constructively or destructively, and this will vary over time. Theoretical bounds for the time duration for which the channel may be assumed to be invariant, known as the \sphinxstyleemphasis{Coherence Time} and denoted \(T_c\), is given by \begin{equation*} \begin{split}T_c =c/v \times 1/f\end{split} \end{equation*} \sphinxAtStartPar where \(c\) is the velocity of the signal, \(v\) is the velocity of the receiver (e.g., moving car or train), and \(f\) is the frequency of the carrier signal that is being modulated. This says the coherence time is inversely proportional to the frequency of the signal and the speed of movement, which makes intuitive sense: The higher the frequency (narrower the wave) the shorter the coherence time, and likewise, the faster the receiver is moving the shorter the coherence time. Based on the target parameters to this model (selected according to the target physical environment), it is possible to calculate \(T_c\), which in turn bounds the rate at which symbols can be transmitted without undue risk of interference. The dynamic nature of the wireless channel is a central challenge to address in the cellular network. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide16}.png} \caption{Received data spread over time due to multipath variation.}\label{\detokenize{radio:id7}}\label{\detokenize{radio:fig-coherence}}\end{figure} \sphinxAtStartPar To complicate matters further, \hyperref[\detokenize{radio:fig-multipath}]{Figure \ref{\detokenize{radio:fig-multipath}}} and \hyperref[\detokenize{radio:fig-coherence}]{\ref{\detokenize{radio:fig-coherence}}} imply the transmission originates from a single antenna, but cell towers are equipped with an array of antennas, each transmitting in a different (but overlapping) direction. This technology, called \sphinxstyleemphasis{Multiple\sphinxhyphen{}Input\sphinxhyphen{}Multiple\sphinxhyphen{}Output (MIMO)}, opens the door to purposely transmitting data from multiple antennas in an effort to reach the receiver, adding even more paths to the environment\sphinxhyphen{}imposed multipath propagation. \sphinxAtStartPar One of the most important consequences of these factors is that the transmitter must receive feedback from every receiver to judge how to best utilize the wireless medium on their behalf. 3GPP specifies a \sphinxstyleemphasis{Channel Quality Indicator (CQI)} for this purpose. In practice, the receiver sends a CQI status report to the base station periodically (e.g., every millisecond). These CQI messages report the observed signal\sphinxhyphen{}to\sphinxhyphen{}noise ratio, which impacts the receiver’s ability to recover the data bits. The base station then uses this information to adapt how it allocates the available radio spectrum to the subscribers it is serving, as well as which coding and modulation scheme to employ. All of these decisions are made by the scheduler. \section{3.2 Scheduler} \label{\detokenize{radio:scheduler}} \sphinxAtStartPar How the scheduler does its job is one of the most important properties of each generation of the cellular network, which in turn depends on the multiplexing mechanism. For example, 2G used \sphinxstyleemphasis{Time Division Multiple Access (TDMA)} and 3G used \sphinxstyleemphasis{Code Division Multiple Access (CDMA)}. How data is multiplexed is also a major differentiator for 4G and 5G, completing the transition from the cellular network being fundamentally circuit\sphinxhyphen{}switched to fundamentally packet\sphinxhyphen{}switched. \sphinxAtStartPar Both 4G and 5G are based on \sphinxstyleemphasis{Orthogonal Frequency\sphinxhyphen{}Division Multiplexing (OFDM)}, an approach that multiplexes data over multiple orthogonal subcarrier frequencies, each of which is modulated independently. One attraction of OFDM is that, by splitting the frequency band into subcarriers, it can send symbols on each subcarrier at a relatively low rate. This makes it easier to correctly decode symbols in the presence of multipath interference. The efficiency of OFDM depends on the selection of subcarrier frequencies so as to avoid interference, that is, how it achieves orthogonality. That topic is beyond the scope of this book. \sphinxAtStartPar As long as you understand that OFDM uses a set of subcarriers, with symbols (each of which contain a few bits of data) being sent at some rate on each subcarrier, that you can appreciate that there are discrete schedulable units of the radio spectrum. The fundamental unit is the time to transmit one symbol on one subcarrier. With that building block in mind, we are now in a position to examine how multiplexing and scheduling work in 4G and 5G. \subsection{3.2.1 Multiplexing in 4G} \label{\detokenize{radio:multiplexing-in-4g}} \sphinxAtStartPar We start with 4G because it provides a foundational understanding that makes 5G easier to explain, where both 4G and 5G use an approach to multiplexing called \sphinxstyleemphasis{Orthogonal Frequency\sphinxhyphen{}Division Multiple Access (OFDMA)}. You can think of OFDMA as a specific application of OFDM. In the 4G case, OFDMA multiplexes data over a set of 12 orthogonal (non\sphinxhyphen{}interfering) subcarrier frequencies, each of which is modulated independently.% \begin{footnote}[1]\sphinxAtStartFootnote 4G uses OFDMA for downstream traffic, and a different multiplexing strategy for upstream transmissions (from user devices to base stations), but we do not describe it because the approach is not applicable to 5G. % \end{footnote} The “Multiple Access” in OFDMA implies that data can simultaneously be sent on behalf of multiple users, each on a different subcarrier frequency and for a different duration of time. The 4G\sphinxhyphen{}defined subbands are narrow (e.g., 15 kHz), and the coding of user data into OFDMA symbols is designed to minimize the risk of data loss due to interference. \sphinxAtStartPar The use of OFDMA naturally leads to conceptualizing the radio spectrum as a 2\sphinxhyphen{}D resource, as shown in \hyperref[\detokenize{radio:fig-sched-grid}]{Figure \ref{\detokenize{radio:fig-sched-grid}}}, with the subcarriers represented in the vertical dimension and the time to transmit symbols on each subcarrier represented in the horizontal dimension. The basic unit of transmission, called a \sphinxstyleemphasis{Resource Element (RE)}, corresponds to a 15\sphinxhyphen{}kHz band around one subcarrier frequency and the time it takes to transmit one OFDMA symbol. The number of bits that can be encoded in each symbol depends on the modulation scheme in use. For example, using \sphinxstyleemphasis{Quadrature Amplitude Modulation (QAM)}, 16\sphinxhyphen{}QAM yields 4 bits per symbol and 64\sphinxhyphen{}QAM yields 6 bits per symbol. The details of the modulation need not concern us here; the key point is that there is a degree of freedom to choose the modulation scheme based on the measured channel quality, sending more bits per symbol (and thus more bits per second) when the quality is high. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide17}.png} \caption{Spectrum abstractly represented by a 2\sphinxhyphen{}D grid of schedulable Resource Elements.}\label{\detokenize{radio:id8}}\label{\detokenize{radio:fig-sched-grid}}\end{figure} \sphinxAtStartPar A scheduler allocates some number of REs to each user that has data to transmit during each 1 ms \sphinxstyleemphasis{Transmission Time Interval (TTI)}, where users are depicted by different colored blocks in \hyperref[\detokenize{radio:fig-sched-grid}]{Figure \ref{\detokenize{radio:fig-sched-grid}}}. The only constraint on the scheduler is that it must make its allocation decisions on blocks of 7x12=84 resource elements, called a \sphinxstyleemphasis{Physical Resource Block (PRB)}. \hyperref[\detokenize{radio:fig-sched-grid}]{Figure \ref{\detokenize{radio:fig-sched-grid}}} shows two back\sphinxhyphen{}to\sphinxhyphen{}back PRBs. Of course time continues to flow along one axis, and depending on the size of the available frequency band (e.g., it might be 100 MHz wide), there may be many more subcarrier slots (and hence PRBs) available along the other axis, so the scheduler is essentially preparing and transmitting a sequence of PRBs. \sphinxAtStartPar Note that OFDMA is not a coding/modulation algorithm, but instead provides a framework for selecting a specific coding and modulation for each subcarrier frequency. QAM is one common example modulation. It is the scheduler’s responsibility to select the modulation to use for each PRB, based on the CQI feedback it has received. The scheduler also selects the coding on a per\sphinxhyphen{}PRB basis, for example, by how it sets the parameters to the turbo code algorithm. \sphinxAtStartPar The 1\sphinxhyphen{}ms TTI corresponds to the time frame in which the scheduler receives feedback from users about the quality of the signal they are experiencing. This is the role of CQI: once every millisecond, each user sends a set of metrics, which the scheduler uses to make its decision as to how to allocate PRBs during the subsequent TTI. \sphinxAtStartPar Another input to the scheduling decision is the \sphinxstyleemphasis{QoS Class Identifier (QCI)}, which indicates the quality\sphinxhyphen{}of\sphinxhyphen{}service each class of traffic is to receive. In 4G, the QCI value assigned to each class (there are twenty six such classes, in total) indicates whether the traffic has a \sphinxstyleemphasis{Guaranteed Bit Rate (GBR)} or not \sphinxstyleemphasis{(non\sphinxhyphen{}GBR)}, plus the class’s relative priority within those two categories. (Note that the 5QI parameter introduced in Chapter 2 serves the same purpose as the QCI parameter in 4G.) \sphinxAtStartPar Finally, keep in mind that \hyperref[\detokenize{radio:fig-sched-grid}]{Figure \ref{\detokenize{radio:fig-sched-grid}}} focuses on scheduling transmissions from a single antenna, but the MIMO technology described above means the scheduler also has to determine which antenna (or more generally, what subset of antennas) will most effectively reach each receiver. But again, in the abstract, the scheduler is charged with allocating a sequence of Resource Elements. \sphinxAtStartPar Note that the scheduler has many degrees of freedom: it has to decide which set of users to service during a given time interval, how many resource elements to allocate to each such user, how to select the coding and modulation levels, and which antenna to transmit their data on. This is an optimization problem that, fortunately, we are not trying to solve here. Our goal is to describe an architecture that allows someone else to design and plug in an effective scheduler. Keeping the cellular architecture open to innovations like this is one of our goals, and as we will see in the next section, becomes even more important in 5G where the scheduler operates with even more degrees of freedom. \subsection{3.2.2 Multiplexing in 5G} \label{\detokenize{radio:multiplexing-in-5g}} \sphinxAtStartPar The transition from 4G to 5G introduces additional flexibility in how the radio spectrum is scheduled, making it possible to adapt the cellular network to a more diverse set of devices and application domains. \sphinxAtStartPar Fundamentally, 5G defines a family of waveforms—unlike LTE, which specified only one waveform—each optimized for a different band in the radio spectrum.% \begin{footnote}[2]\sphinxAtStartFootnote A waveform is defined by the frequency, amplitude, and phase\sphinxhyphen{}shift independent property (shape) of a signal. A sine wave is a simple example of a waveform. % \end{footnote} The bands with carrier frequencies below 1 GHz are designed to deliver mobile broadband and massive IoT services with a primary focus on range. Carrier frequencies between 1\sphinxhyphen{}6 GHz are designed to offer wider bandwidths, focusing on mobile broadband and mission\sphinxhyphen{}critical applications. Carrier frequencies above 24 GHz (mmWaves) are designed to provide super\sphinxhyphen{}wide bandwidths over short, line\sphinxhyphen{}of\sphinxhyphen{}sight coverage. \sphinxAtStartPar These different waveforms affect the scheduling and subcarrier intervals (i.e., the “size” of the resource elements described in the previous section). \begin{itemize} \item {} \sphinxAtStartPar For frequency range 1 (410 MHz \sphinxhyphen{} 7125 MHz), 5G allows maximum 100 MHz bandwidths. In this case, there are three waveforms with subcarrier spacings of 15, 30 and 60 kHz. (We used 15 kHz in the example shown in \hyperref[\detokenize{radio:fig-sched-grid}]{Figure \ref{\detokenize{radio:fig-sched-grid}}}.) The corresponding to scheduling intervals of 0.5, 0.25, and 0.125 ms, respectively. (We used 0.5 ms in the example shown in \hyperref[\detokenize{radio:fig-sched-grid}]{Figure \ref{\detokenize{radio:fig-sched-grid}}}.) \item {} \sphinxAtStartPar For millimeter bands, also known as frequency range 2 (24.25 GHz \sphinxhyphen{} 52.6 GHz), bandwidths may go from 50 MHz up to 400 MHz. There are two waveforms, with subcarrier spacings of 60 kHz and 120 kHz. Both have scheduling intervals of 0.125 ms. \end{itemize} \sphinxAtStartPar These various configurations of subcarrier spacing and scheduling intervals are sometimes called the \sphinxstyleemphasis{numerology} of the radio’s air interface. \sphinxAtStartPar This range of numerology is important because it adds another degree of freedom to the scheduler. In addition to allocating radio resources to users, it has the ability to dynamically adjust the size of the resource by changing the waveform being used. With this additional freedom, fixed\sphinxhyphen{}sized REs are no longer the primary unit of resource allocation. We instead use more abstract terminology, and talk about allocating \sphinxstyleemphasis{Resource Blocks} to subscribers, where the 5G scheduler determines both the size and number of Resource Blocks allocated during each time interval. \sphinxAtStartPar \hyperref[\detokenize{radio:fig-scheduler}]{Figure \ref{\detokenize{radio:fig-scheduler}}} depicts the role of the scheduler from this more abstract perspective. Just as with 4G, CQI feedback from the receivers and the 5QI quality\sphinxhyphen{}of\sphinxhyphen{}service class selected by the subscriber are the two key pieces of input to the scheduler. Note that the set of 5QI values available in 5G is considerably greater than its QCI counterpart in 4G, reflecting the increasing differentiation among classes that 5G aims to support. For 5G, each class includes the following attributes: \begin{itemize} \item {} \sphinxAtStartPar Resource Type: Guaranteed Bit Rate (GBR), Delay\sphinxhyphen{}Critical GBR, Non\sphinxhyphen{}GBR \item {} \sphinxAtStartPar Priority Level \item {} \sphinxAtStartPar Packet Delay Budget \item {} \sphinxAtStartPar Packet Error Rate \item {} \sphinxAtStartPar Maximum Data Burst \item {} \sphinxAtStartPar Averaging Window \end{itemize} \sphinxAtStartPar Note that while the preceding discussion could be interpreted to imply a one\sphinxhyphen{}to\sphinxhyphen{}one relationship between subscribers and a 5QI, it is more accurate to say that each 5QI is associated with a class of traffic (often corresponding to some type of application), where a given subscriber might be sending and receiving traffic that belongs to multiple classes at any given time. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide18}.png} \caption{Scheduler allocates Resource Blocks to user data streams based on CQI feedback from receivers and the 5QI parameters associated with each class of service.}\label{\detokenize{radio:id9}}\label{\detokenize{radio:fig-scheduler}}\end{figure} \section{3.3 Virtualized Scheduler (Slicing)} \label{\detokenize{radio:virtualized-scheduler-slicing}} \sphinxAtStartPar The discussion up to this point presumes a single scheduler is suitable for all workloads, but different applications have different requirements for how their traffic gets scheduled. For example, some applications care about latency and others care more about bandwidth. \sphinxAtStartPar While in principle one could define a sophisticated scheduler that takes dozens of different factors into account, 5G has introduced a mechanism that allows the underlying resources (in this case radio spectrum) to be “sliced” between different uses. The key to slicing is to add a layer of indirection between the scheduler and the physical resource blocks. Slicing, like much of 5G, has received a degree of hype, but it boils down to virtualization at the level of the radio scheduler. \sphinxAtStartPar As shown in \hyperref[\detokenize{radio:fig-hypervisor}]{Figure \ref{\detokenize{radio:fig-hypervisor}}}, the idea is to first schedule offered traffic demands to virtual RBs, and then perform a second mapping of Virtual RBs to Physical RBs. This sort of virtualization is common in resource allocators throughout computing systems because we want to separate how many resources are allocated to each user (or virtual machine in the computing case) from the decision as to which physical resources are actually assigned. This virtual\sphinxhyphen{}to\sphinxhyphen{}physical mapping is performed by a layer typically known as a \sphinxstyleemphasis{Hypervisor}, and the important thing to keep in mind is that it is totally agnostic about which user’s segment is affected by each translation. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide19}.png} \caption{Wireless Hypervisor mapping virtual resource blocks to physical resource blocks.}\label{\detokenize{radio:id10}}\label{\detokenize{radio:fig-hypervisor}}\end{figure} \sphinxAtStartPar Having decoupled the Virtual RBs from Physical RBs, it is now possible to define multiple Virtual RB sets (of varying sizes), each with its own scheduler. \hyperref[\detokenize{radio:fig-multi-sched}]{Figure \ref{\detokenize{radio:fig-multi-sched}}} gives an example with two equal\sphinxhyphen{}sized RB sets. The important consequence is this: having made the macro\sphinxhyphen{}decision that the Physical RBs are divided into two equal partitions, the scheduler associated with each partition is free to allocate Virtual RBs independently from the other. For example, one scheduler might be designed to deal with high\sphinxhyphen{}bandwidth video traffic and another scheduler might be optimized for low\sphinxhyphen{}latency IoT traffic. Alternatively, a certain fraction of the available capacity could be reserved for premium customers or other high\sphinxhyphen{}priority traffic (e.g., public safety), with the rest shared among everyone else. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide20}.png} \caption{Multiple schedulers running on top of wireless hypervisor.}\label{\detokenize{radio:id11}}\label{\detokenize{radio:fig-multi-sched}}\end{figure} \sphinxAtStartPar A final point to note is that there is considerable flexibility in the allocation of resources to slices. While the example above shows resources allocated in a fixed manner to each slice, it is possible to make unused resources in one slice available to another slice, as long as they can be reclaimed when needed. This is similar to how work\sphinxhyphen{}conserving scheduling works in network queues: resources are not wasted if the offered load in a class is less than the rate guaranteed for that class. \section{3.4 New Use Cases} \label{\detokenize{radio:new-use-cases}} \sphinxAtStartPar We conclude by noting that up to this point we have described 5G as introducing additional degrees of freedom into how data is scheduled for transmission, but when taken as a whole, the end result is a qualitatively more powerful radio. This new 5G air interface specification, which is commonly referred to as \sphinxstyleemphasis{New Radio (NR)}, enables new use cases that go well beyond simply delivering increased bandwidth. 3GPP defined five such use cases: \begin{itemize} \item {} \sphinxAtStartPar Enhanced Mobile Broadband (eMBB) \item {} \sphinxAtStartPar Ultra\sphinxhyphen{}Reliable Low\sphinxhyphen{}Latency Communications (URLLC) \item {} \sphinxAtStartPar Massive Internet of Things (MIoT) \item {} \sphinxAtStartPar Vehicle to Infrastructure/Vehicle (V2X) \item {} \sphinxAtStartPar High\sphinxhyphen{}Performance Machine\sphinxhyphen{}Type Communications (HMTC) \end{itemize} \sphinxAtStartPar These use cases reflect the requirements introduced in Chapter 1, and can be attributed to four fundamental improvements in how 5G multiplexes data onto the radio spectrum. \sphinxAtStartPar The first improvement is being able to change the waveform. This effectively introduces the ability to dynamically change the size and number of schedulable resource units, which opens the door to making fine\sphinxhyphen{}grained scheduling decisions that are critical to predictable, low\sphinxhyphen{}latency communication. \sphinxAtStartPar The second is related to the “Multiple Access” aspect of how distinct traffic sources are multiplexed onto the available spectrum. In 4G, multiplexing happens in both the frequency and time domains for downstream traffic, but only in the frequency domain for upstream traffic. 5G NR multiplexes both upstream and downstream traffic in both the time and frequency domains. Doing so provides finer\sphinxhyphen{}grained scheduling control needed by latency\sphinxhyphen{}sensitive applications. \sphinxAtStartPar The third advance is related to the new spectrum available to 5G NR, with the mmWave allocations opening above 24 GHz being especially important. This is not only because of the abundance of capacity—which makes it possible to set aside dedicated capacity for applications that require low\sphinxhyphen{}latency communication—but also because the higher frequency enables even finer\sphinxhyphen{}grained resource blocks (e.g., scheduling intervals as short as 0.125 ms). Again, this improves scheduling granularity to the benefit of applications that cannot tolerate unpredictable latency. \sphinxAtStartPar The fourth is related to delivering mobile connectivity to a massive number of IoT devices, ranging from devices that require mobility support and modest data rates (e.g. wearables, asset trackers) to devices that support intermittent transmission of a few bytes of data (e.g., sensors, meters). None of these devices are particularly latency\sphinxhyphen{}sensitive or bandwidth\sphinxhyphen{}hungry, but they often require long battery lifetimes, and hence, reduced hardware complexity that draws less power. \sphinxAtStartPar Support for IoT device connectivity revolves around allocating some of the available radio spectrum to a light\sphinxhyphen{}weight (simplified) air interface. This approach started with Release 13 of LTE via two complementary technologies: mMTC and NB\sphinxhyphen{}IoT (NarrowBand\sphinxhyphen{}IoT). Both technologies build on a significantly simplified version of LTE—i.e., limiting the numerology and flexibility needed to achieve high spectrum utilization—so as to allow for simpler IoT hardware design. mMTC delivers up to 1 Mbps over 1.4 MHz of bandwidth and NB\sphinxhyphen{}IoT delivers a few tens of kbps over 200 kHz of bandwidth; hence the term \sphinxstyleemphasis{NarrowBand}. Both technologies have been designed to support over 1 million devices per square kilometer. With Release 16, both technologies can be operated in\sphinxhyphen{}band with 5G, but still based on LTE numerology. Starting with Release 17, a simpler version of 5G NR, called \sphinxstyleemphasis{NR\sphinxhyphen{}Light}, will be introduced as the evolution of mMTC. NR\sphinxhyphen{}Light is expected to scale the device density even further. \sphinxAtStartPar As a complement to these four improvements, 5G NR is designed to support partitioning the available bandwidth, with different partitions dynamically allocated to different classes of traffic (e.g., high\sphinxhyphen{}bandwidth, low\sphinxhyphen{}latency, and low\sphinxhyphen{}complexity). This is the essence of \sphinxstyleemphasis{slicing}, as discussed above. Moreover, once traffic with different requirements can be served by different slices, 5G NR’s approach to multiplexing is general enough to support varied scheduling decisions for those slices, each tailored for the target traffic. We return to the applications of slicing in Chapter 6. \chapter{Chapter 4: Radio Access Network} \label{\detokenize{ran:chapter-4-radio-access-network}}\label{\detokenize{ran::doc}} \sphinxAtStartPar The high\sphinxhyphen{}level description of the RAN in Chapter 2 was mostly silent about the RAN’s internal structure. We now focus on those details, and in doing so, explain how the RAN is being transformed in 5G. \sphinxAtStartPar You can think of the RAN as having one main job: to transfer packets between the mobile core and a set of UEs. This means it is deeply involved in the management and scheduling of radio spectrum that we discussed in the last chapter, but there is more to it than that. We start by describing the stages in the RAN’s packet processing pipeline, and then showing how these stages are being disaggregated, distributed, and implemented. \sphinxAtStartPar Note that the deconstruction of the RAN presented in this chapter represents a combination of standardized specifications and implementation strategies. The former continues to be under the purview of the 3GPP, but the latter are primarily influenced by a second organization: the \sphinxstyleemphasis{Open\sphinxhyphen{}RAN Alliance (O\sphinxhyphen{}RAN)} introduced in Chapter 1. O\sphinxhyphen{}RAN is led by network operators with the goal of developing a software\sphinxhyphen{}based implementation of the RAN that eliminates vendor lock\sphinxhyphen{}in. Such business forces are certainly a factor in where 5G mobile networks are headed, but our goal in this chapter is to identify the technical design decisions involved in that evolution. \section{4.1 Packet Processing Pipeline} \label{\detokenize{ran:packet-processing-pipeline}} \sphinxAtStartPar \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} shows the packet processing stages historically bundled in base stations, as specified by the 3GPP standard. Note that the figure depicts the base station as a pipeline (running left\sphinxhyphen{}to\sphinxhyphen{}right for packets sent to the UE) but it is equally valid to view it as a protocol stack (as is typically done in official 3GPP documents). Also note that (for now) we are agnostic as to how these stages are implemented. Since we are ultimately heading towards a cloud\sphinxhyphen{}based implementation, one possible implementation strategy would be a microservice per box. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide110}.png} \caption{RAN processing pipeline, including both user and control plane components.}\label{\detokenize{ran:id5}}\label{\detokenize{ran:fig-pipeline}}\end{figure} \sphinxAtStartPar The key stages are as follows. \begin{itemize} \item {} \sphinxAtStartPar RRC (Radio Resource Control) \(\rightarrow\) Responsible for configuring the coarse\sphinxhyphen{}grained and policy\sphinxhyphen{}related aspects of the pipeline. The RRC runs in the RAN’s control plane; it does not process packets in the user plane. \item {} \sphinxAtStartPar PDCP (Packet Data Convergence Protocol) \(\rightarrow\) Responsible for compressing and decompressing IP headers, ciphering and deciphering, integrity protection and integrity verification, duplication, reordering and in\sphinxhyphen{}order delivery, and out\sphinxhyphen{}of\sphinxhyphen{}order delivery. \item {} \sphinxAtStartPar RLC (Radio Link Control) \(\rightarrow\) Responsible for segmentation and reassembly, as well as reliably transmitting/receiving segments using error correction through ARQ (automatic repeat request). \item {} \sphinxAtStartPar MAC (Media Access Control) \(\rightarrow\) Responsible for buffering, multiplexing and demultiplexing segments, including all real\sphinxhyphen{}time scheduling decisions about what segments are transmitted when. Also able to make a “late” forwarding decision (i.e., to alternative carrier frequencies, including Wi\sphinxhyphen{}Fi). \item {} \sphinxAtStartPar PHY (Physical Layer) \(\rightarrow\) Responsible for coding and modulation (as discussed in Chapter 3), including FEC. \end{itemize} \sphinxAtStartPar The last two stages in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} (D/A conversion and the RF front\sphinxhyphen{}end) are beyond the scope of this book. \sphinxAtStartPar While it is simplest to view the stages in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} as a pure left\sphinxhyphen{}to\sphinxhyphen{}right pipeline, the Scheduler described in Section 3.2 (denoted “S” in the figure) runs in the MAC stage/layer, and implements the “main loop” for outbound traffic: It reads data from the upstream RLC and schedules transmissions to the downstream PHY. Since the Scheduler determines the number of bytes to transmit to a given UE during each time period (based on all the factors outlined in Chapter 3), it must request (get) a segment of that length from the upstream queue. In practice, the size of the segment that can be transmitted on behalf of a single UE during a single scheduling interval can range from a few bytes to an entire IP packet. \sphinxAtStartPar Also note that a combination of the RRC and PDCP are responsible for the observation made in Section 2.3: that a \sphinxstyleemphasis{“base station can be viewed as a specialized forwarder”}. The control plane logic that decides whether this base station should continue processing a packet or forward it to another base station runs in the RRC, and the corresponding user plane mechanism that carries out the forwarding decision is implemented in the PDCP. The interface between these two elements is defined by the 3GPP spec, but the decision\sphinxhyphen{}making logic is an implementation choice (and historically proprietary). This control logic is generically referred as the \sphinxstyleemphasis{Radio Resource Management (RRM)}, not to be confused with the standards\sphinxhyphen{}defined RRC stage depicted in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}}. \section{4.2 Split RAN} \label{\detokenize{ran:split-ran}} \sphinxAtStartPar The next step is to understand how the functionality outlined above is partitioned between physical elements, and hence, “split” across centralized and distributed locations. The dominant option has historically been “no split,” with the entire pipeline shown in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} running in the base station. Going forward, the 3GPP standard has been extended to allow for multiple split\sphinxhyphen{}points, with the partition shown in \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}} being actively pursued by the operator\sphinxhyphen{}led O\sphinxhyphen{}RAN Alliance. It is the split we adopt throughout the rest of this book. Note that the split between centralized and distributed components mirrors the split made in SDN, with similar motivations. We discuss further how SDN techniques are applied to the RAN below. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide24}.png} \caption{Split RAN processing pipeline distributed across a Central Unit (CU), Distributed Unit (DU), and Radio Unit (RU).}\label{\detokenize{ran:id6}}\label{\detokenize{ran:fig-split-ran}}\end{figure} \sphinxAtStartPar This results in a RAN\sphinxhyphen{}wide configuration similar to that shown in \hyperref[\detokenize{ran:fig-ran-hierarchy}]{Figure \ref{\detokenize{ran:fig-ran-hierarchy}}}, where a single \sphinxstyleemphasis{Central Unit (CU)} running in the cloud serves multiple \sphinxstyleemphasis{Distributed Units (DUs)}, each of which in turn serves multiple \sphinxstyleemphasis{Radio Units (RUs)}. Critically, the RRC (centralized in the CU) is responsible for making only near\sphinxhyphen{}real\sphinxhyphen{}time configuration and control decisions, while the Scheduler that is part of the MAC stage is responsible for all real\sphinxhyphen{}time scheduling decisions. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide32}.png} \caption{Split RAN hierarchy, with one CU serving multiple DUs, each of which serves multiple RUs.}\label{\detokenize{ran:id7}}\label{\detokenize{ran:fig-ran-hierarchy}}\end{figure} \sphinxAtStartPar Because scheduling decisions for radio transmission are made by the MAC layer in real time, a DU needs to be “near” (within 1ms) the RUs it manages. (You can’t afford to make scheduling decisions based on out\sphinxhyphen{}of\sphinxhyphen{}date channel information.) One familiar configuration is to co\sphinxhyphen{}locate a DU and an RU in a cell tower. But when an RU corresponds to a small cell, many of which might be spread across a modestly\sphinxhyphen{}sized geographic area (e.g., a mall, campus, or factory), then a single DU would likely service multiple RUs. The use of mmWave in 5G is likely to make this latter configuration all the more common. \sphinxAtStartPar Also note that the Split RAN changes the nature of the Backhaul Network, which originally connected the base stations back to the Mobile Core. With the Split RAN there are multiple connections, which are officially labeled as follows. \begin{itemize} \item {} \sphinxAtStartPar RU\sphinxhyphen{}DU connectivity is called the Fronthaul \item {} \sphinxAtStartPar DU\sphinxhyphen{}CU connectivity is called the Midhaul \item {} \sphinxAtStartPar CU\sphinxhyphen{}Mobile Core connectivity is called the Backhaul \end{itemize} \sphinxAtStartPar For more insight into design considerations for interconnecting the distributed components of a Split RAN, we recommend the NGMN Alliance Report. \phantomsection\label{\detokenize{ran:reading-backhaul}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://www.ngmn.org/wp-content/uploads/NGMN\_RANEV\_D4\_BH\_FH\_Evolution\_V1.01.pdf}{RAN Evolution Project: Backhaul and Fronthaul Evolution}. NGMN Alliance Report, March 2015. \end{sphinxadmonition} \sphinxAtStartPar One observation about the CU (which becomes relevant in Chapter 6 when we incorporate it into a managed cloud service) is that one might co\sphinxhyphen{}locate the CU and Mobile Core in the same cluster, meaning the backhaul is implemented in the cluster switching fabric. In such a configuration, the midhaul then effectively serves the same purpose as the original backhaul, and the fronthaul is constrained by the predictable/low\sphinxhyphen{}latency requirements of the MAC stage’s real\sphinxhyphen{}time scheduler. This situation is further complicated by the fact that the mobile core itself may be disaggregated, as discussed in Chapter 5. \sphinxAtStartPar A second observation about the CU shown in \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}} is that it encompasses two functional blocks—the RRC and the PDCP—which lie on the RAN’s control plane and user plane, respectively. This separation is consistent with the idea of CUPS introduced in Chapter 2, and plays an increasingly important role as we dig deeper into how the RAN is implemented. For now, we note that the two parts are sometimes referred to as the CU\sphinxhyphen{}CP and CU\sphinxhyphen{}UP, respectively. \sphinxAtStartPar We conclude our description of the split RAN architecture with the alternative depiction in \hyperref[\detokenize{ran:fig-split-alt}]{Figure \ref{\detokenize{ran:fig-split-alt}}}. For completeness, this figure identifies the standardized interfaces between the components (e.g., N2, N3, F1\sphinxhyphen{}U, F1\sphinxhyphen{}C, and Open Fronthaul). We’re not going to talk about these interfaces, except to note that they exist and there is a corresponding 3GPP specification that spells out the details. Instead, we’re going to comment on the availability of open source implementations for each component. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=150\sphinxpxdimen]{{Slide101}.png} \caption{Alternative depiction of the Split RAN components, showing the 3GPP\sphinxhyphen{}specified inter\sphinxhyphen{}unit interfaces.}\label{\detokenize{ran:id8}}\label{\detokenize{ran:fig-split-alt}}\end{figure} \sphinxAtStartPar With respect to the Central Unit, most of the complexity is in the CU\sphinxhyphen{}CP, which, as we’ll see in the next section, is being re\sphinxhyphen{}engineered using SDN, with open source solutions available. With respect to the Radio Unit, nearly all the complexity is in D/A conversion and how the resulting analog signal is amplified. Incumbent vendors have significant proprietary know\sphinxhyphen{}how in this space, which will almost certainly remain closed. \sphinxAtStartPar With respect to the Distributed Unit, the news is mixed, and correspondingly, the figure shows more detail. The High\sphinxhyphen{}PHY module—which corresponds to all but the RF modulation step of \hyperref[\detokenize{radio:fig-modulation}]{Figure \ref{\detokenize{radio:fig-modulation}}} in Section 3.1—is one of the most complex components in the RAN stack. An open source implementation of the High\sphinxhyphen{}PHY, known as FlexRAN, exists and is widely used in commercial products. The only caveat is that the software license restricts usage to Intel processors, although it is also the case that the FlexRAN software exploits Intel\sphinxhyphen{}specific hardware capabilities. As for the rest of the DU, the MAC is the other source of high\sphinxhyphen{}value closed technology, particularly in how scheduling is done. There is an open source version made available by the Open Air Initiative (OAI), but its usage is restricted to research\sphinxhyphen{}only deployments. \phantomsection\label{\detokenize{ran:reading-du-impl}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://www.intel.com/content/www/us/en/developer/topic-technology/edge-5g/tools/flexran.html}{FlexRAN: Reference Architecture for Wireless Access}. \sphinxAtStartPar \sphinxhref{https://openairinterface.org/}{Open Air Interface}. \end{sphinxadmonition} \section{4.3 Software\sphinxhyphen{}Defined RAN} \label{\detokenize{ran:software-defined-ran}} \sphinxAtStartPar We now describe how the RAN is being implemented according to SDN principles, resulting in an SD\sphinxhyphen{}RAN. The key architectural insight is shown in \hyperref[\detokenize{ran:fig-rrc-split}]{Figure \ref{\detokenize{ran:fig-rrc-split}}}, where the RRC from \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} is partitioned into two sub\sphinxhyphen{}components: the one on the left provides a 3GPP\sphinxhyphen{}compliant way for the RAN to interface to the Mobile Core’s control plane (the figure labels this sub\sphinxhyphen{}component as a “Proxy”), while the one on the right opens a new programmatic API for exerting software\sphinxhyphen{}based control over the pipeline that implements the RAN user plane. \sphinxAtStartPar To be more specific, the left sub\sphinxhyphen{}component simply forwards control packets between the Mobile Core and the PDCP, providing a path over which the Mobile Core can communicate with the UE for control purposes, whereas the right sub\sphinxhyphen{}component implements the core of the RRC’s control functionality (which as we explained in Section 4.1 is also known as RRM). This latter component is commonly referred to as the \sphinxstyleemphasis{RAN Intelligent Controller (RIC)} in O\sphinxhyphen{}RAN architecture documents, so we adopt this terminology. The “Near\sphinxhyphen{}Real Time” qualifier indicates the RIC is part of 10\sphinxhyphen{}100ms control loop implemented in the CU, as opposed to the \textasciitilde{}1ms control loop required by the MAC scheduler running in the DU. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide42}.png} \caption{RRC disaggregated into a Mobile Core facing control plane component (a proxy) and a Near\sphinxhyphen{}Real\sphinxhyphen{}Time Controller.}\label{\detokenize{ran:id9}}\label{\detokenize{ran:fig-rrc-split}}\end{figure} \sphinxAtStartPar Although not shown in \hyperref[\detokenize{ran:fig-rrc-split}]{Figure \ref{\detokenize{ran:fig-rrc-split}}}, keep in mind (from \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}}) that the RRC and the PDCP form the CU. Reconciling these two figures is a little bit messy, but to a first approximation, the PDCP corresponds to the CU\sphinxhyphen{}UP and RRC\sphinxhyphen{}Proxy corresponds to the CU\sphinxhyphen{}CP, with the RIC “lifted out” and responsible for overseeing both. We postpone a diagram depicting this relationship until Section 4.5, where we summarize the end\sphinxhyphen{}to\sphinxhyphen{}end result. For now, the important takeaway is that the SDN\sphinxhyphen{}inspired refactoring of the RAN is free both to move functionality around and to introduce new module boundaries, as long as the original 3GPP\sphinxhyphen{}defined interfaces are preserved. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide52}.png} \caption{Example set of control applications (xApps) running on top of Near\sphinxhyphen{}Real\sphinxhyphen{}Time RAN Controller (RIC), controlling a distributed set of Split RAN elements (CU, DU, RU).}\label{\detokenize{ran:id10}}\label{\detokenize{ran:fig-ran-controller}}\end{figure} \sphinxAtStartPar Completing the picture, \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}} shows the Near\sphinxhyphen{}RT RIC implemented as an SDN Controller hosting a set of SDN control apps. The RIC maintains a \sphinxstyleemphasis{RAN Network Information Base (R\sphinxhyphen{}NIB)}—a common set of information that can be consumed by numerous control apps. The R\sphinxhyphen{}NIB includes time\sphinxhyphen{}averaged CQI values and other per\sphinxhyphen{}session state (e.g., GTP tunnel IDs, 5QI values for the type of traffic), while the MAC (as part of the DU) maintains the instantaneous CQI values required by the real\sphinxhyphen{}time scheduler. More generally, the R\sphinxhyphen{}NIB includes the following state: \begin{itemize} \item {} \sphinxAtStartPar Fixed Nodes (RU/DU/CU Attributes) \begin{itemize} \item {} \sphinxAtStartPar Identifiers \item {} \sphinxAtStartPar Version \item {} \sphinxAtStartPar Config Report \item {} \sphinxAtStartPar RRM config \item {} \sphinxAtStartPar PHY resource usage \end{itemize} \item {} \sphinxAtStartPar Mobile Nodes (UE Attributes) \begin{itemize} \item {} \sphinxAtStartPar Devices \begin{itemize} \item {} \sphinxAtStartPar Identifiers \item {} \sphinxAtStartPar Capability \item {} \sphinxAtStartPar Measurement Config \item {} \sphinxAtStartPar State (Active/Idle) \end{itemize} \item {} \sphinxAtStartPar Links (\sphinxstyleemphasis{Actual} and \sphinxstyleemphasis{Potential}) \begin{itemize} \item {} \sphinxAtStartPar Identifiers \item {} \sphinxAtStartPar Link Type \item {} \sphinxAtStartPar Config/Bearer Parameters \item {} \sphinxAtStartPar 5QI Value \end{itemize} \end{itemize} \item {} \sphinxAtStartPar Virtual Constructs (Slice Attributes) \begin{itemize} \item {} \sphinxAtStartPar Links \item {} \sphinxAtStartPar Bearers/Flows \item {} \sphinxAtStartPar Validity Period \item {} \sphinxAtStartPar Desired KPIs \item {} \sphinxAtStartPar MAC RRM Configuration \item {} \sphinxAtStartPar RRM Control Configuration \end{itemize} \end{itemize} \sphinxAtStartPar The four example Control Apps (xApps) in \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}} do not constitute an exhaustive list, but they do represent the sweet spot for SDN, with its emphasis on central control over distributed forwarding. These functions—Link Aggregation Control, Interference Management, Load Balancing, and Handover Control—are often implemented by individual base stations with only local visibility, but they have global consequences. The SDN approach is to collect the available input data centrally, make a globally optimal decision, and then push the respective control parameters back to the base stations for execution. Evidence using an analogous approach to optimize wide\sphinxhyphen{}area networks over many years (see, for example, B4) is compelling. \phantomsection\label{\detokenize{ran:reading-b4}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For an example of how SDN principles have been successfully applied to a production network, we recommend \sphinxhref{https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf}{B4: Experience with a Globally\sphinxhyphen{}Deployed Software Defined WAN}. ACM SIGCOMM, August 2013. \end{sphinxadmonition} \sphinxAtStartPar One way to characterize xApps is based on the current practice of controlling the mobile link at two different levels. At a fine\sphinxhyphen{}grained level, per\sphinxhyphen{}node and per\sphinxhyphen{}link control are conducted using the RRM functions that are distributed across the individual base stations.% \begin{footnote}[1]\sphinxAtStartFootnote Pedantically, Radio Resource Management (RRM) is another name for the collection of control functionality typically implemented in the RRC stage of the RAN pipeline. % \end{footnote} RRM functions include scheduling, handover control, link and carrier aggregation control, bearer control, and access control. At a coarse\sphinxhyphen{}grained level, regional mobile network optimization and configuration is conducted using \sphinxstyleemphasis{Self\sphinxhyphen{}Organizing Network (SON)} functions. These functions oversee neighbor lists, manage load balancing, optimize coverage and capacity, aim for network\sphinxhyphen{}wide interference mitigation, manage centrally configured parameters, and so on. As a consequence of these two levels of control, it is not uncommon to see reference to \sphinxstyleemphasis{RRM Applications} and \sphinxstyleemphasis{SON Applications}, respectively, in O\sphinxhyphen{}RAN documents for SD\sphinxhyphen{}RAN. For example, the Interference Management and Load Balancing xApps in \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}} are SON Applications, while the other two xApps are RRM Applications. \sphinxAtStartPar Keep in mind, however, that this characterization of xApps is based on past (pre\sphinxhyphen{}SDN) implementations of the RAN. This is helpful as the industry transitions to SD\sphinxhyphen{}RAN, but the situation is likely to change. SDN is transforming the RAN, so new ways of controlling the RAN—resulting in applications that do not fit neatly into the RRM vs SON dichotomy—can be expected to emerge over time. \section{4.4 Near Real\sphinxhyphen{}Time RIC} \label{\detokenize{ran:near-real-time-ric}} \sphinxAtStartPar Drilling down to the next level of detail, \hyperref[\detokenize{ran:fig-ric}]{Figure \ref{\detokenize{ran:fig-ric}}} shows an exemplar implementation of a RIC based on a retargeting of the Open Network OS (ONOS) for the SD\sphinxhyphen{}RAN use case. ONOS (described in our SDN book) was originally designed to support traditional wireline network switches using standard interfaces (OpenFlow, P4Runtime, etc.). For the SD\sphinxhyphen{}RAN use case, the ONOS\sphinxhyphen{}based RIC instead supports a set of RAN\sphinxhyphen{}specific north\sphinxhyphen{} and south\sphinxhyphen{}facing interfaces, but internally takes advantage of the same collection of subsystems (microservices) as in the wireline case.% \begin{footnote}[2]\sphinxAtStartFootnote Technically, the O\sphinxhyphen{}RAN definition of the RIC refers to the combination of xApps and the underlying platform (in our case ONOS), but we emphasize the distinction between the two, in keeping with the SDN model of distinguishing between the Network OS and the suite of Control Apps that run on it. % \end{footnote} \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide62}.png} \caption{O\sphinxhyphen{}RAN compliant RAN Intelligent Controller (RIC) built by adapting and extending ONOS.}\label{\detokenize{ran:id11}}\label{\detokenize{ran:fig-ric}}\end{figure} \sphinxAtStartPar Specifically, the ONOS\sphinxhyphen{}based RIC includes a Topology Service to keep track of the fixed RAN infrastructure, a Device Service to track the UEs, and a Configuration Service to manage RAN\sphinxhyphen{}wide configuration state. All three of these services are implemented as Kubernetes\sphinxhyphen{}based microservices, and take advantage of a scalable key\sphinxhyphen{}value store. \sphinxAtStartPar Of the three interfaces called out in \hyperref[\detokenize{ran:fig-ric}]{Figure \ref{\detokenize{ran:fig-ric}}}, the \sphinxstylestrong{A1} and \sphinxstylestrong{E2} interfaces are based on pre\sphinxhyphen{}existing 3GPP standards. The third, \sphinxstylestrong{xApp SDK}, is specific to the ONOS\sphinxhyphen{}based implementation. The O\sphinxhyphen{}RAN Alliance is using it to drive towards a unified API (and corresponding SDK) for building RIC\sphinxhyphen{}agnostic xApps. \sphinxAtStartPar The A1 interface provides a means for the mobile operator’s management plane—typically called the \sphinxstyleemphasis{OSS/BSS (Operations Support System / Business Support System)} in the Telco world—to configure the RAN. We briefly introduced the OSS/BSS in Section 2.5, but all you need to know about it for our purposes is that such a component sits at the top of all Telco software stacks. It is the source of all configuration settings and business logic needed to operate a network. You can think of A1 as the RAN’s counterpart to gNMI/gNOI (gRPC Network Management Interface/gRPC Network Operations Interface), a pair of configuration APIs commonly used to configure commodity cloud hardware. \sphinxAtStartPar The Near\sphinxhyphen{}RT RIC uses the E2 interface to control the underlying RAN elements, including the CU, DUs, and RUs. A requirement of the E2 interface is that it be able to connect the Near\sphinxhyphen{}RT RIC to different types of RAN elements from different vendors. This range is reflected in the API, which revolves around a \sphinxstyleemphasis{Service Model} abstraction. The idea is that each RAN element advertises a Service Model, which effectively defines the set of RAN Functions the element is able to support. The RIC then issues a combination of the following four operations against this Service Model. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Report:} RIC asks the element to report a function\sphinxhyphen{}specific value setting. \item {} \sphinxAtStartPar \sphinxstylestrong{Insert:} RIC instructs the element to activate a user plane function. \item {} \sphinxAtStartPar \sphinxstylestrong{Control:} RIC instructs the element to activate a control plane function. \item {} \sphinxAtStartPar \sphinxstylestrong{Policy:} RIC sets a policy parameter on one of the activated functions. \end{itemize} \sphinxAtStartPar Of course, it is the RAN element, through its published Service Model, that defines the relevant set of functions that can be activated, the variables that can be reported, and policies that can be set. The O\sphinxhyphen{}RAN community is working on three vendor\sphinxhyphen{}agnostic Service Models. The first, called \sphinxstyleemphasis{Key Performance Measurement} (abbreviated \sphinxstyleemphasis{E2SM\sphinxhyphen{}KPM}), specifies the metrics that can be retrieved from RAN elements. The second, called \sphinxstyleemphasis{RAN Control} (abbreviated \sphinxstyleemphasis{E2SM\sphinxhyphen{}RC}), specifies parameters that can be set to control RAN elements. The third, called \sphinxstyleemphasis{Cell Configuration and Control} (abbreviated \sphinxstyleemphasis{E2SM\sphinxhyphen{}CCC}), exposes configuration parameters at the cell level. \sphinxAtStartPar In simple terms, E2SM\sphinxhyphen{}KPM defines what values can be \sphinxstyleemphasis{read} and E2SM\sphinxhyphen{}RC and E2SM\sphinxhyphen{}CCC defines what values can be \sphinxstyleemphasis{written}. Because the available values can be highly variable across all possible devices, we can expect different vendors will support only a subset of the entire collection. This will limit the “universality” the O\sphinxhyphen{}RAN was hoping to achieve in an effort to break vendor lock\sphinxhyphen{}in, but that outcome is familiar to network operators who have been dealing with divergent \sphinxstyleemphasis{Management Information Bases (MIBs)} since the earliest days of the Internet. \sphinxAtStartPar Finally, the xApp SDK, which is specific to the ONOS\sphinxhyphen{}based implementation, is currently little more than a “pass through” of the E2 interface. This implies the xApps are expected to be aware of the available Service Models. One of the challenges the SDK has to deal with is how data passed to/from the RAN elements is encoded. For historical reasons, the E2 interface uses ASN.1 formatting, whereas the ONOS\sphinxhyphen{}RIC internally uses gRPC and Protocol Buffers to communicate between the set of microservices. The south\sphinxhyphen{}bound E2 interface in \hyperref[\detokenize{ran:fig-ric}]{Figure \ref{\detokenize{ran:fig-ric}}} translates between the two formats. The SDK currently makes the gRPC\sphinxhyphen{}based API available to xApps. \phantomsection\label{\detokenize{ran:reading-onos}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar To learn more about the details of ONOS and its interfaces, we recommend the chapter in our SDN book that covers it in depth. \sphinxhref{https://sdn.systemsapproach.org/onos.html}{Software\sphinxhyphen{}Defined Networks: A Systems Approach. Chapter 6: Network OS}. \end{sphinxadmonition} \section{4.5 Control Loops} \label{\detokenize{ran:control-loops}} \sphinxAtStartPar We conclude this description of RAN internals by revisiting the sequence of steps involved in disaggregation, which, as the previous three sections reveal, is being pursued in multiple tiers. In doing so, we tie up several loose ends, and focus attention on the resulting three control loops. \sphinxAtStartPar In the first tier of disaggregation, 3GPP defines multiple options for how the RAN can be split and distributed, with the pipeline shown in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} disaggregated into the independently operating CU, DU, and RU components shown in \hyperref[\detokenize{ran:fig-disagg1}]{Figure \ref{\detokenize{ran:fig-disagg1}}}. The O\sphinxhyphen{}RAN Alliance has selected specific disaggregation options from 3GPP and is developing open interfaces between these components. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide72}.png} \caption{First tier of RAN disaggregation: Split RAN.}\label{\detokenize{ran:id12}}\label{\detokenize{ran:fig-disagg1}}\end{figure} \sphinxAtStartPar The second tier of disaggregation focuses on the control/user plane separation (CUPS) of the CU, resulting in the CU\sphinxhyphen{}UP and CU\sphinxhyphen{}CP shown in \hyperref[\detokenize{ran:fig-disagg2}]{Figure \ref{\detokenize{ran:fig-disagg2}}}. The control plane in question is the 3GPP control plane, where the CU\sphinxhyphen{}UP realizes a pipeline for user traffic and the CU\sphinxhyphen{}CP focuses on control message signaling between Mobile Core and the disaggregated RAN components (as well as to the UE). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide82}.png} \caption{Second tier of RAN disaggregation: CUPS.}\label{\detokenize{ran:id13}}\label{\detokenize{ran:fig-disagg2}}\end{figure} \sphinxAtStartPar The third tier follows the SDN paradigm by separating most of RAN control (RRC functions) from the disaggregated RAN components, and logically centralizing them as applications running on an SDN Controller, which corresponds to the Near\sphinxhyphen{}RT RIC shown previously in \hyperref[\detokenize{ran:fig-rrc-split}]{Figures \ref{\detokenize{ran:fig-rrc-split}}} and \hyperref[\detokenize{ran:fig-ran-controller}]{\ref{\detokenize{ran:fig-ran-controller}}}. This SDN\sphinxhyphen{}based disaggregation is repeated in \hyperref[\detokenize{ran:fig-ctl-loops}]{Figure \ref{\detokenize{ran:fig-ctl-loops}}}, which also shows the O\sphinxhyphen{}RAN\sphinxhyphen{}prescribed interfaces A1 and E2 introduced in the previous section. (Note that all the edges in \hyperref[\detokenize{ran:fig-disagg1}]{Figures \ref{\detokenize{ran:fig-disagg1}}} and \hyperref[\detokenize{ran:fig-disagg2}]{\ref{\detokenize{ran:fig-disagg2}}} correspond to 3GPP\sphinxhyphen{}defined interfaces, as identified in Section 4.2, but their details are outside the scope of this discussion.) \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=800\sphinxpxdimen]{{Slide92}.png} \caption{Third tier of RAN disaggregation: SDN.}\label{\detokenize{ran:id14}}\label{\detokenize{ran:fig-ctl-loops}}\end{figure} \sphinxAtStartPar Taken together, the A1 and E2 interfaces complete two of the three major control loops of the RAN: the outer (non\sphinxhyphen{}real\sphinxhyphen{}time) loop has the Non\sphinxhyphen{}RT RIC as its control point and the middle (near\sphinxhyphen{}real\sphinxhyphen{}time) loop has the Near\sphinxhyphen{}RT RIC as its control point. The third (innermost) control loop—shown in \hyperref[\detokenize{ran:fig-ctl-loops}]{Figure \ref{\detokenize{ran:fig-ctl-loops}}} running inside the DU—includes the real\sphinxhyphen{}time Scheduler embedded in the MAC stage of the RAN pipeline. The two outer control loops have rough time bounds of \textgreater{}\textgreater{}1sec and \textgreater{}10ms, respectively. As we saw in Chapter 2, the real\sphinxhyphen{}time control loop is assumed to be \textless{}1ms. \sphinxAtStartPar This raises the question of how specific functionality is distributed between the Non\sphinxhyphen{}RT RIC, Near\sphinxhyphen{}RT RIC, and DU. Starting with the second pair (i.e., the two inner loops), it is the case that not all RRC functions can be centralized; some need to be implemented in the DU. The SDN\sphinxhyphen{}based disaggregation then focuses on those that can be centralized, with the Near\sphinxhyphen{}RT RIC supporting the RRC applications and the SON applications mentioned in Section 4.3. \sphinxAtStartPar Turning to the outer two control loops, the Near RT\sphinxhyphen{}RIC opens the possibility of introducing policy\sphinxhyphen{}based RAN control, whereby interrupts (exceptions) to operator\sphinxhyphen{}defined policies would signal the need for the outer loop to become involved. For example, one can imagine developing learning\sphinxhyphen{}based controls, where the inference engines for these controls would run as applications on the Near RT\sphinxhyphen{}RIC, and their non\sphinxhyphen{}real\sphinxhyphen{}time learning counterparts would run elsewhere. The Non\sphinxhyphen{}RT RIC would then interact with the Near\sphinxhyphen{}RT RIC to deliver relevant operator policies from the Management Plane to the Near RT\sphinxhyphen{}RIC over the A1 interface. \chapter{Chapter 5: Mobile Core} \label{\detokenize{core:chapter-5-mobile-core}}\label{\detokenize{core::doc}} \sphinxAtStartPar The Mobile Core provides IP connectivity to the RAN. It authenticates UEs as they connect, tracks them as they move from one base station to another, ensures that this connectivity fulfills the promised QoS requirements, and meters usage for billing. \sphinxAtStartPar Historically, all of these functions were provided by one or more proprietary network appliances. But like the rest of the 5G mobile network, these appliances are being disaggregated and implemented as a set of cloud services, with the goal of improving feature velocity for new classes of applications. It is also the case that as the range of use cases grows more diverse, a one\sphinxhyphen{}size\sphinxhyphen{}fits\sphinxhyphen{}all approach will become problematic. The expectation is that it should be possible to customize and specialize the Mobile Core on a per\sphinxhyphen{}application basis. \sphinxAtStartPar This chapter introduces the functional elements of the Mobile Core, and describes different strategies for implementing that functionality. \section{5.1 Identity Management} \label{\detokenize{core:identity-management}} \sphinxAtStartPar There are two equally valid views of the Mobile Core. The Internet\sphinxhyphen{}centric view is that each local instantiation of the Mobile Core (e.g., serving a metro area) acts as a router that connects a physical RAN (one of many possible access network technologies, not unlike WiFi) to the global Internet. In this view, IP addresses serve as the unique global identifier that makes it possible for any RAN\sphinxhyphen{}connected device to communicate with any Internet\sphinxhyphen{}addressable device or service. The 3GPP\sphinxhyphen{}centric view is that a distributed set of Mobile Cores (interconnected by one or more backbone technologies, of which the Internet is just one example) cooperate to turn a set of physical RANs into one logically global RAN. In this perspective, the IMSI burned into a device’s SIM card serves as the global identifier that makes it possible for any two mobile devices to communicate with each other. \sphinxAtStartPar Both of these perspectives are correct, but since broadband communication using Internet protocols to access cloud services is the dominant use case, this section takes an Internet\sphinxhyphen{}centric perspective of the Mobile Core. But before getting to that, we first need to understand several things about the 3GPP\sphinxhyphen{}centric perspective. \sphinxAtStartPar For starters, we need to be aware of the distinction between “identity” and “identifier”. The first term is commonly used when talking about principals or users, and the second term is used when talking about abstract objects or physical devices. Unfortunately, the two terms are conflated in the 3GPP architecture: The acronym IMSI explicitly includes the word “Identity”, where the “S” in both IMSI and SIM stands for subscriber (a kind of principal), yet the IMSI is also used as a global identifier for a UE connected to the mobile network. This conflation breaks down when there could be tens or hundreds of IoT devices for every customer, with no obvious association among them. Accounting for this problem is an “architecture alignment” fix we discuss in the next chapter when we describe how to provide Private 5G Connectivity as a managed cloud service. \sphinxAtStartPar If we take the view that an IMSI is primarily a global identifier for UEs, then we can think of it as the mobile network’s equivalent of a 48\sphinxhyphen{}bit 802.3 or 802.11 MAC address. This includes how addresses are assigned to ensure uniqueness: (MCC, MNC) pairs are assigned by a global authority to every MNO, each of which then decides how to uniquely assign the rest of the IMSI identifier space to devices. This approach is similar to how network vendors are assigned a unique prefix for all the MAC addresses they configure into the NIC cards and WiFi chips they ship, but with one big difference: It is the MNO, rather than the vendor, that is responsible for assigning IMSIs to SIM cards. This makes the IMSI allocation problem closer to how the Internet assigns IP addresses to end hosts, but unlike DHCP, the IMSI\sphinxhyphen{}to\sphinxhyphen{}device binding is static. \sphinxAtStartPar This is important because, unlike 802.11 addresses, IMSIs are also intended to support global routing. (Here, we are using a liberal notion of routing—to locate an object—and focusing on the original 3GPP\sphinxhyphen{}perspective of the global RAN in which the Internet is just a possible packet network that interconnects Mobile Cores.) A hierarchically distributed database maps IMSIs onto the collection of information needed to forward data to the corresponding device. This includes a combination of relatively \sphinxstyleemphasis{static} information about the level of service the device expects to receive (including the corresponding phone number and subscriber profile/account information), and more \sphinxstyleemphasis{dynamic} information about the current location of the device (including which Mobile Core, and which base station served by that Core, currently connects the device to the global RAN). \sphinxAtStartPar This mapping service has a name, or rather, several names that keep changing from generation to generation. In 2G and 3G it was called HLR (Home Location Registry). In 4G the HLR maintains only static information and a separate HSS (Home Subscriber Server) maintains the more dynamic information. In 5G the HLR is renamed the UDR (Unified Data Registry) and the HSS is renamed UDM (Unified Data Management). We will see the UDM in Section 5.2 because of the role it plays \sphinxstyleemphasis{within} a single instance of the Mobile Core. \sphinxAtStartPar There are, of course, many more details to the process—including how to find a device that has roamed to another MNO’s network—but conceptually the process is straightforward. (As a thought experiment, imagine how you would build a “logically global WiFi” using just 802.11 addresses, rather than depending on the additional layer of addressing provided by IP.) The important takeaway is that IMSIs are used to locate the Mobile Core instance that is then responsible for authenticating the device, tracking the device as it moves from base station to base station within that Core’s geographic region, and forwarding packets to/from the device. \sphinxAtStartPar Two additional observations about the relationship between IMSIs and IP addresses are worth highlighting. First, the odds of someone trying to “call” or “text” an IoT device, drone, camera, or robot are virtually zero. It is the IP address assigned to each device (by the local Mobile Core) that is used to locate (route packets to) the device. In this context, the IMSI plays exactly the same role in a physical RAN as an 802.11 address plays in a LAN, and the Mobile Core behaves just like any access router. \sphinxAtStartPar Second, whether a device connects to a RAN or some other access network, it is automatically assigned a new IP address any time it moves from one coverage domain to another. Even for voice calls in the RAN case, ongoing calls are dropped whenever a device moves between instantiations of the Mobile Core (i.e., uninterrupted mobility is supported only within the region served by a given Core). This is typically not a problem when the RAN is being used to deliver broadband connectivity because Internet devices are almost always clients \sphinxstyleemphasis{requesting} a cloud service; they just start issuing requests with their new (dynamically assigned) IP address. \section{5.2 Functional Components} \label{\detokenize{core:functional-components}} \sphinxAtStartPar The 5G Mobile Core, which 3GPP calls the \sphinxstyleemphasis{5GC}, adopts a microservice\sphinxhyphen{}like architecture officially known as the 3GPP \sphinxstyleemphasis{Service Based Architecture}. We say “microservice\sphinxhyphen{}like” because while the 3GPP specification spells out this level of disaggregation, it is really just describing a set of functional blocks and not prescribing an implementation. In practice, a set of functional blocks is very different from the collection of engineering decisions that go into designing a microservice\sphinxhyphen{}based system. That said, viewing the collection of components shown in \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}} as a set of microservices is a reasonable working model (for now). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide211}.png} \caption{5G Mobile Core (5GC), represented as a collection of microservices, where 3GPP defines the interfaces connecting the Mobile Core CP and UP to the RAN (denoted N2 and N3, respectively).}\label{\detokenize{core:id3}}\label{\detokenize{core:fig-5g-core}}\end{figure} \sphinxAtStartPar Starting with the User Plane (UP), the \sphinxstyleemphasis{UPF (User Plane Function)} forwards traffic between the RAN and the Internet. In addition to IP packet forwarding, the UPF is responsible for policy enforcement, lawful intercept, traffic usage measurement, and QoS policing. These are all common functions in access routers, even if they go beyond what you usually find in enterprise or backbone routers. The other detail of note is that, because the RAN is an overlay network, the UPF is responsible for tunneling (i.e., encapsulating and decapsulating) packets as they are transmitted to and from base stations over the N3 interface (as depicted in \hyperref[\detokenize{arch:fig-tunnels}]{Figure \ref{\detokenize{arch:fig-tunnels}}} of Section 2.3). \sphinxAtStartPar The rest of the functional elements in \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}} implement the Control Plane (CP). Of these, two represent the majority of the functionality that’s unique to the Mobile Core CP (as sketched in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} of Section 2.4): \begin{itemize} \item {} \sphinxAtStartPar \sphinxstyleemphasis{AMF (Access and Mobility Management Function):} Responsible for connection and reachability management, mobility management, access authorization, and location services. \item {} \sphinxAtStartPar \sphinxstyleemphasis{SMF (Session Management Function):} Manages each UE session, including IP address allocation, selection of associated UP function, control aspects of QoS, and control aspects of UP routing. \end{itemize} \sphinxAtStartPar In other words, the AMF authorizes access when a UE first connects to one of the local base stations, and then tracks (but does not control) which base station currently serves each UE. The SMF then allocates an IP address to each AMF\sphinxhyphen{}authorized UE, and directly interacts with the UPF to maintain per\sphinxhyphen{}device session state. \sphinxAtStartPar Of particular note, the per\sphinxhyphen{}UE session state controlled by the SMF (and implemented by the UPF) includes a packet buffer in which packets destine to an idle UE are queued during the time the UE transitions to active state. This feature was originally designed to avoid data loss during a voice call, but its value is less obvious when the data is an IP packet since end\sphinxhyphen{}to\sphinxhyphen{}end protocols like TCP are prepared to retransmit lost packets. On the other hand, if idle\sphinxhyphen{}to\sphinxhyphen{}active transitions are too frequent, they can be problematic for TCP. \sphinxAtStartPar Before continuing with our inventory of control\sphinxhyphen{}related elements in \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}}, it is important to note we show only a fraction of the full set that 3GPP defines. The full set includes a wide range of possible features, many of which are either speculative (i.e., identify potential functionality) or overly prescriptive (i.e., identify well\sphinxhyphen{}known cloud native microservices). We limit our discussion to functional elements that provide value in the private 5G deployments that we focus on. Of these, several provide functionality similar to what one might find in any microservice\sphinxhyphen{}based application: \begin{itemize} \item {} \sphinxAtStartPar \sphinxstyleemphasis{AUSF (Authentication Server Function):} Authenticates UEs. \item {} \sphinxAtStartPar \sphinxstyleemphasis{UDM (Unified Data Management):} Manages user identity, including the generation of authentication credentials and access authorization. \item {} \sphinxAtStartPar \sphinxstyleemphasis{UDR (Unified Data Repository):} Manages user static subscriber\sphinxhyphen{}related information. \item {} \sphinxAtStartPar \sphinxstyleemphasis{UDSF (Unstructured Data Storage Function):} Used to store unstructured data, and so is similar to a \sphinxstyleemphasis{key\sphinxhyphen{}value store}. \item {} \sphinxAtStartPar \sphinxstyleemphasis{NEF (Network Exposure Function):} Exposes select capabilities to third\sphinxhyphen{}party services, and so is similar to an \sphinxstyleemphasis{API Server}. \item {} \sphinxAtStartPar \sphinxstyleemphasis{NRF (Network Repository Function):} Used to discover available services (network functions), and so is similar to a \sphinxstyleemphasis{Discovery Service}. \end{itemize} \sphinxAtStartPar The above list includes 3GPP\sphinxhyphen{}specified control functions that are, in some cases, similar to well\sphinxhyphen{}known microservices. In such cases, substituting an existing cloud native component is a viable implementation option. For example, MongoDB can be used to implement a UDSF. In other cases, however, such a one\sphinxhyphen{}for\sphinxhyphen{}one swap is not possible due to assumptions 3GPP makes. For example, AUSF, UDM, UDR, and AMF collectively implement a \sphinxstyleemphasis{Authentication and Authorization Service}, but an option like OAuth2 could not be used in their place because (a) UDM and UDR are assumed to be part of the global identity mapping service discussed in Section 5.1, and (b) 3GPP specifies the interface by which the various components request service from each other (e.g., AMF connects to the RAN via the N2 interface depicted in \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}}). We will see how to cope with such issues in Section 5.3, where we talk about implementation issues in more detail. \sphinxAtStartPar Finally, \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}} shows two other functional elements that export a northbound interface to the management plane (not shown): \begin{itemize} \item {} \sphinxAtStartPar \sphinxstyleemphasis{PCF (Policy Control Function):} Manages the policy rules for the rest of the Mobile Core CP. \item {} \sphinxAtStartPar \sphinxstyleemphasis{NSSF (Network Slice Selection Function):} Manages how network slices are selected to serve a given UE. \end{itemize} \sphinxAtStartPar Keep in mind that even though 3GPP does not directly prescribe a microservice implementation, the overall design clearly points to a cloud native solution as the desired end\sphinxhyphen{}state for the Mobile Core. Of particular note, introducing a distinct storage service means that all the other services can be stateless, and hence, more readily scalable. \section{5.3 Control Plane} \label{\detokenize{core:control-plane}} \sphinxAtStartPar This section describes two different strategies for implementing the Mobile Core CP. Both correspond to open source projects that are available for download and experimentation. \subsection{5.3.1 SD\sphinxhyphen{}Core} \label{\detokenize{core:sd-core}} \sphinxAtStartPar Our first example, called SD\sphinxhyphen{}Core, is a nearly one\sphinxhyphen{}for\sphinxhyphen{}one translation of the functional blocks shown in \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}} into a cloud native implementation. A high\sphinxhyphen{}level schematic is shown in \hyperref[\detokenize{core:fig-sd-core}]{Figure \ref{\detokenize{core:fig-sd-core}}}, where each element corresponds to a scalable set of Kubernetes\sphinxhyphen{}hosted containers. We include this schematic even though it looks quite similar to \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}} because it highlights four implementation details. \phantomsection\label{\detokenize{core:reading-sd-core}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://opennetworking.org/sd-core/}{SD\sphinxhyphen{}Core}. \end{sphinxadmonition} \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide22}.png} \caption{SD\sphinxhyphen{}Core implementation of the Mobile Core Control Plane, including support for Standalone (SA) deployment of both 4G and 5G.}\label{\detokenize{core:id4}}\label{\detokenize{core:fig-sd-core}}\end{figure} \sphinxAtStartPar First, SD\sphinxhyphen{}Core supports both the 5G and 4G versions of the Mobile Core,% \begin{footnote}[1]\sphinxAtStartFootnote SD\sphinxhyphen{}Core’s 4G Core is a fork of the OMEC project and its 5G Core is a fork of the Free5GC project. % \end{footnote} which share a common User Plane (UPF). We have not discussed details of the 4G Core, but observe that it is less disaggregated. In particular, the components in the 5G Core are specified so that they can be stateless, simplifying the task of horizontally scaling them out as load dictates. (The rough correspondence between 4G and 5G is: MME\sphinxhyphen{}to\sphinxhyphen{}AMF, SPGW\_C\sphinxhyphen{}to\sphinxhyphen{}SMF, HSS\sphinxhyphen{}to\sphinxhyphen{}UDM, and PCRF\sphinxhyphen{}to\sphinxhyphen{}PCF.) Although not shown in the schematic, there is also a scalable key\sphinxhyphen{}value store microservice based on MongoDB. It is used to make Core\sphinxhyphen{}related state persistent for the Control Planes; for example, UDM/UDR (5G) and HSS (4G) write subscriber state to MongoDB. \sphinxAtStartPar Second, \hyperref[\detokenize{core:fig-sd-core}]{Figure \ref{\detokenize{core:fig-sd-core}}} illustrates 3GPP’s \sphinxstyleemphasis{Standalone (SA)} deployment option, in which 4G and 5G networks co\sphinxhyphen{}exist and run independently. They share a UPF implementation, but UPF instances are instantiated separately for each RAN/Core pair, with support for both the 4G and 5G interfaces, denoted \sphinxstyleemphasis{S1\sphinxhyphen{}U} and \sphinxstyleemphasis{N3}, respectively. Although not obvious from the SA example, 3GPP defines an alternative transition plan, called \sphinxstyleemphasis{NSA (Non\sphinxhyphen{}Standalone)}, in which separate 4G and 5G RANs were paired with either a 4G Core or a 5G Core. The details of how that works are not relevant to this discussion, except to make the point that production networks almost never get to enjoy a “flag day” on which a new version is universally substituted for an old version. A migration plan has to be part of the design. More information on this topic can be found in a GSMA Report. \phantomsection\label{\detokenize{core:reading-migration}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://www.gsma.com/futurenetworks/wp-content/uploads/2018/04/Road-to-5G-Introduction-and-Migration\_FINAL.pdf}{Road to 5G: Introduction and Migration}. GSMA Report, April 2018. \end{sphinxadmonition} \sphinxAtStartPar Third, \hyperref[\detokenize{core:fig-sd-core}]{Figure \ref{\detokenize{core:fig-sd-core}}} shows many of the 3GPP\sphinxhyphen{}defined inter\sphinxhyphen{}component interfaces. These include an over\sphinxhyphen{}the\sphinxhyphen{}air interface between base stations and UEs (\sphinxstyleemphasis{NR Uu}), control interfaces between the Core and both UEs and base stations (\sphinxstyleemphasis{N1} and \sphinxstyleemphasis{N2}, respectively), a user plane interface between the Core and base stations (\sphinxstyleemphasis{N3}), and a data plane interface between the Core and the backbone network (\sphinxstyleemphasis{N6}). \sphinxAtStartPar The schematic also shows interfaces between the individual microservices that make up the Core’s Control Plane; for example, \sphinxstyleemphasis{Nudm} is the interface to the UDM microservice. These latter interfaces are RESTful, meaning clients access each microservice by issuing GET, PUT, POST, PATCH, and DELETE operations over HTTP, where a service\sphinxhyphen{}specific schema defines the available resources that can be accessed. Note that some of these interfaces are necessary for interoperability (e.g., \sphinxstyleemphasis{N1} and \sphinxstyleemphasis{N Uu} make it possible to connect your phone to any MNO’s network), but others could be seen as internal implementation details. We’ll see how Magma takes advantage of this distinction in the next section. \sphinxAtStartPar Fourth, by adopting a cloud native design, SD\sphinxhyphen{}Core benefits from being able to horizontally scale individual microservices. But realizing this benefit isn’t always straightforward. In particular, because the AMF is connected to the RAN by SCTP (corresponding to the \sphinxstyleemphasis{N1} and \sphinxstyleemphasis{N2} interfaces shown in \hyperref[\detokenize{core:fig-sd-core}]{Figure \ref{\detokenize{core:fig-sd-core}}}), it is necessary to put an \sphinxstyleemphasis{SCTP load balancer} in front of the AMF. This load balancer terminates the SCTP connections, and distributes requests across a set of AMF containers. These AMF instances then depend on a scalable backend store (specifically MongoDB) to read and write shared state. \subsection{5.3.2 Magma} \label{\detokenize{core:magma}} \sphinxAtStartPar Magma is an open source Mobile Core implementation that takes a different and slightly non\sphinxhyphen{}standard approach. Magma is similar to SD\sphinxhyphen{}Core in that it is implemented as a set of microservices, but it differs in that it is designed to be particularly suitable for remote and rural environments with poor backhaul connectivity. This emphasis, in turn, leads Magma to (1) adopt an SDN\sphinxhyphen{}inspired approach to how it separates functionality into centralized and distributed components, and (2) factor the distributed functionality into microservices without strict adherence to all the standard 3GPP interface specifications. This refactoring is also a consequence of Magma being designed to unify 4G, 5G, and WiFi under a single architecture. \sphinxAtStartPar One of the first things to note about Magma is that it takes a different view of “backhaul” from the approaches we have seen to date. Whereas the backhaul networks shown previously connect the eNBs/gNBs and radio towers back to the mobile core (\hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}), Magma actually puts much of the mobile core functionality right next to the radio as seen in \hyperref[\detokenize{core:fig-magma-peru}]{Figure \ref{\detokenize{core:fig-magma-peru}}}. It is able to do this because of the way it splits the core into centralized and distributed parts. So Magma views “backhaul” as the link that connects a remote deployment to the rest of the Internet (including the central components), contrasting with conventional 3GPP usage. As explored further below, this can overcome many of the challenges that unreliable backhaul links introduce in conventional approaches. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide112}.png} \caption{Overall architecture of the Magma Mobile Core, including support for 4G and 5G, and Wi\sphinxhyphen{}Fi. There is one central Orchestrator and typically many Access Gateways (AGWs).}\label{\detokenize{core:id5}}\label{\detokenize{core:fig-magma-arch}}\end{figure} \sphinxAtStartPar \hyperref[\detokenize{core:fig-magma-arch}]{Figure \ref{\detokenize{core:fig-magma-arch}}} shows the overall Magma architecture. The central part of Magma is the single box in the figure marked \sphinxstyleemphasis{Central Control \& Management (Orchestrator)}. This is roughly analogous to the central controller found in typical SDN systems, and provides a northbound API by which an operator or other software systems (such as a traditional OSS/BSS) can interact with the Magma core. The orchestrator communicates over backhaul links with Access Gateways (AGWs), which are the distributed components of Magma. A single AGW typically handles a small number of eNBs/gNBs. As an example, see \hyperref[\detokenize{core:fig-magma-peru}]{Figure \ref{\detokenize{core:fig-magma-peru}}} which includes a single eNB and AGW located on a radio tower. In this example, a point\sphinxhyphen{}to\sphinxhyphen{}point wireless link is used for backhaul. \sphinxAtStartPar The AGW is designed to have a small footprint, so that small deployments do not require a datacenter’s worth of equipment. Each AGW also contains both data plane and control plane elements. This is a little different from the classic approach to SDN systems in which only the data plane is distributed. Magma can be described as a hierarchical SDN approach, as the control plane itself is divided into a centralized part (running in the Orchestrator) and a distributed part (running in the AGW). \hyperref[\detokenize{core:fig-magma-arch}]{Figure \ref{\detokenize{core:fig-magma-arch}}} shows the distributed control plane components and data plane in detail. We postpone a general discussion of orchestration until Chapter 6. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=300\sphinxpxdimen]{{peru_deploy_labelled}.jpg} \caption{A sample Magma deployment in rural Peru, showing (a) point\sphinxhyphen{}to\sphinxhyphen{}point wireless backhaul, (b) LTE radio and antenna, (c) ruggedized embedded PC serving as AGW, and (d) solar power and battery backup for site.}\label{\detokenize{core:id6}}\label{\detokenize{core:fig-magma-peru}}\end{figure} \sphinxAtStartPar Magma differs from the standard 3GPP approach in that it terminates 3GPP protocols logically close to the edge, which in this context corresponds to two interface points: (1) the radio interface connecting Magma to an eNB or gNB (implemented by set of modules on the left side of the AGW in the figure) or the federation interface connecting Magma to another mobile network (implemented by the \sphinxstyleemphasis{Federation Gateway} module in the figure). Everything “between” those two external interfaces is free to deviate from the 3GPP specification, which has a broad impact as discussed below. \sphinxAtStartPar One consequence of this approach is that Magma can interoperate with other implementations \sphinxstyleemphasis{only} at the edges. Thus, it is possible to connect a Magma mobile core to any standards\sphinxhyphen{}compliant 4G or 5G base station and expect it to work, and similarly, it is possible to federate a Magma core with an existing MNO’s 4G or 5G network. However, since Magma does not implement all the 3GPP interfaces that are internal to a mobile packet core, it is not possible to arbitrarily mix and match components within the core. Whereas (in principle) a traditional 3GPP implementation would permit an AMF from one vendor to interoperate with the SMF of another vendor, it is not possible to connect parts of a mobile core from another vendor (or another open source project) with parts of Magma, aside from via the two interfaces just described. \sphinxAtStartPar Being free to deviate from the 3GPP spec means Magma can take a unifying approach across multiple wireless technologies, including 4G, 5G and WiFi. There is a set of functions that the core must implement for any radio technology (e.g., finding the appropriate policy for a given subscriber by querying a database); Magma provides them in an access\sphinxhyphen{}technology\sphinxhyphen{}independent way. These functions form the heart of an Access Gateway (AGW), as illustrated on the right side of \hyperref[\detokenize{core:fig-magma-arch}]{Figure \ref{\detokenize{core:fig-magma-arch}}}. On the other hand, control protocols that are specific to a given radio technology are terminated in technology\sphinxhyphen{}specific modules close to the radio. For example, \sphinxstyleemphasis{SCTP} shown on the left side of the figure is the RAN tunneling protocol introduced in Section 2.3. These technology\sphinxhyphen{}specific modules then communicate with the generic functions (e.g., subscriber management, access control and management) on the right using gRPC messages that are technology\sphinxhyphen{}agnostic. \sphinxAtStartPar Magma’s design is particularly well suited for environments where backhaul links are unreliable, for example, when a satellite is used. This is because the 3GPP protocols that traditionally have to traverse the backhaul from core to eNB/gNB are quite sensitive to loss and latency. Loss or latency can cause connections to be dropped, which in turn forces UEs to repeat the process of attaching to the core. In practice, not all UEs handle this elegantly, sometimes ending up in a “stuck” state. \sphinxAtStartPar Magma addresses the challenge of unreliable backhaul in several ways. First, Magma frequently avoids sending messages over the backhaul entirely by running more functionality in the AGW, which is located close to the radio as seen above. Functions that would be centralized in a conventional 3GPP implementation are distributed out to the access gateways in Magma. Thus, for example, the operations required to authenticate and attach a UE to the core can typically be completed using information cached locally in the AGW, without any traffic crossing the backhaul. Secondly, when Magma does need to pass information over a backhaul link (e.g., to obtain configuration state from the orchestrator), it does so using gRPC, which is designed to operate reliably in the face of unreliable or high\sphinxhyphen{}latency links. \sphinxAtStartPar Note that while Magma has distributed much of the control plane out to the AGWs, it still supports centralized management via the Orchestrator. For example, adding a new subscriber to the network is done centrally, and the relevant AGW then obtains the necessary state to authenticate that subscriber when their UE tries to attach to the network. \sphinxAtStartPar Finally, Magma adopts a \sphinxstyleemphasis{desired state} model for managing runtime and configuration state. By this we mean that it communicates a state change (e.g., the addition of a new session in the user plane) by specifying the desired end state via an API call. This is in contrast with the \sphinxstyleemphasis{incremental update} model that is common in the 3GPP specification. When the desired end state is communicated, the loss of a message or failure of a component has less serious consequences. This makes reasoning about changes across elements of the system more robust in the case of partial failures, which are common in challenged environments like the ones Magma is designed to serve. \sphinxAtStartPar Consider an example where we are establishing user plane state for a set of active sessions. Initially, there are two active sessions, X and Y. Then a third UE becomes active and a session Z needs to be established. In the incremental update model, the control plane would instruct the user plane to “add session Z”. The desired state model, by contrast, communicates the entire new state: “the set of sessions is now X, Y, Z”. The incremental model is brittle in the face of failures. If a message is lost, or a component is temporarily unable to receive updates, the receiver falls out of sync with the sender. So it is possible that the control plane believes that sessions X, Y and Z have been established, while the user plane has state for only X and Y. By sending the entire desired end state, Magma ensures that the receiver comes back into sync with the sender once it is able to receive messages again. \sphinxAtStartPar As described, this approach might appear inefficient because it implies sending complete state information rather than incremental updates. However, at the scale of an AGW, which handles on the order of hundreds to a few thousands of subscribers, it is possible to encode the state efficiently enough to overcome this drawback. With the benefit of experience, mechanisms have been added to Magma to avoid overloading the orchestrator, which has state related to all subscribers in the network. \sphinxAtStartPar The desired state approach is hardly novel but differs from typical 3GPP systems. It allows Magma to tolerate occasional communication failures or component outages due to software restarts, hardware failures, and so on. Limiting the scope of 3GPP protocols to the very edge of the network is what enables Magma to rethink the state synchronization model. The team that worked on Magma describes their approach in more detail in an NSDI paper. \phantomsection\label{\detokenize{core:reading-magma}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar S. Hasan, \sphinxstyleemphasis{et al.} \sphinxhref{https://arxiv.org/abs/2209.10001}{Building Flexible, Low\sphinxhyphen{}Cost Wireless Access Networks With Magma}. NSDI, April 2023. \end{sphinxadmonition} \sphinxAtStartPar Finally, while we have focused on its Control Plane, Magma also includes a User Plane component. The implementation is fairly simple, and is based on Open vSwitch (OVS). Having a programmable user plane is important, as it needs to support a range of access technologies, and at the same time, OVS meets the performance needs of AGWs. However, this choice of user plane is not fundamental to Magma, and other implementations have been considered. We take a closer look at the User Plane in the next section. \section{5.4 User Plane} \label{\detokenize{core:user-plane}} \sphinxAtStartPar The User Plane of the Mobile Core—corresponding to the UPF component in \hyperref[\detokenize{core:fig-5g-core}]{Figure \ref{\detokenize{core:fig-5g-core}}}—connects the RAN to the Internet. Much like the data plane for any router, the UPF forwards IP packets, but because UEs often sleep to save power and may be in the process of being handed off from one base station to another, it sometimes has to buffer packets for an indeterminate amount of time. Also like other routers, a straightforward way to understand the UPF is to think of it as implementing a collection of Match/Action rules, where the UPF first classifies each packet against a set of matching rules, and then executes the associated action. \sphinxAtStartPar Using 3GPP terminology, packet classification is defined by a set of \sphinxstyleemphasis{Packet Detection Rules (PDRs)}, where a given PDR might simply match the device’s IP address, but may also take the domain name of the far end\sphinxhyphen{}point into consideration. Each attached UE has at least two PDRs, one for uplink traffic and one for downlink traffic, plus possibly additional PDRs to support multiple traffic classes (e.g., for different QoS levels, pricing plans, and so on.). The Control Plane creates, updates, and removes PDRs as UEs attach, move, and detach. \sphinxAtStartPar Each PDR then identifies one or more actions to execute, which in 3GPP terminology are also called “rules”, of which there are four types: \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Forwarding Action Rules (FARs):} Instructs the UPF to forward downlink packets to a particular base station and uplink traffic to a next\sphinxhyphen{}hop router. Each FAR specifies a set of parameters needed to forward the packet (e.g., how to tunnel downlink packets to the appropriate base station), plus one of the following processing flags: a \sphinxtitleref{forward} flag indicates the packet should be forwarded up to the Internet; a \sphinxtitleref{tunnel} flag indicates the packet should be tunneled down to a base station; a \sphinxtitleref{buffer} flag indicates the packet should be buffered until the UE becomes active; and a \sphinxtitleref{notify} flag indicates that the CP should be notified to awaken an idle UE. FARs are created and removed when a device attaches or detaches, respectively, and the downlink FAR changes the processing flag when the device moves, goes idle, or awakes. \item {} \sphinxAtStartPar \sphinxstylestrong{Buffering Action Rules (BARs):} Instructs the UPF to buffer downlink traffic for idle UEs, while also sending a \sphinxtitleref{Downlink Data Notification} to the Control Plane. This notification, in turn, causes the CP to instruct the base station to awaken the UE. Once the UE becomes active, the UPF releases the buffered traffic and resumes normal forwarding. The buffering and notification functions are activated by modifying a FAR to include \sphinxtitleref{buffer} and \sphinxtitleref{notify} flags, as just described. An additional set of parameters are used to configure the buffer, for example setting its maximum size (number of bytes) and duration (amount of time). Optionally, the CP can itself buffer packets by creating a PDR that directs the UPF to forward data packets to the control plane. \item {} \sphinxAtStartPar \sphinxstylestrong{Usage Reporting Rules (URRs):} Instructs the UPF to periodically send usage reports for each UE to the CP. These reports include counts of the packets sent/received for uplink/downlink traffic for each UE and traffic class. These reports are used to both limit and bill subscribers. The CP creates and removes URRs when the device attaches and detaches, respectively, and each URR specifies whether usage reports should be sent periodically or when a quota is exceeded. A UE typically has two URRs (for uplink/downlink usage), but if a subscriber’s plan includes special treatment for certain types of traffic, an additional URR is created for each such traffic class. \item {} \sphinxAtStartPar \sphinxstylestrong{Quality Enforcement Rules (QERs):} Instructs the UPF to guarantee a minimum amount of bandwidth and to enforce a bandwidth cap. These parameters are specified on a per\sphinxhyphen{}UE / per\sphinxhyphen{}direction / per\sphinxhyphen{}class basis. The CP creates and removes QERs when a device attaches and detaches, respectively, and modifies them according to operator\sphinxhyphen{}defined events, such as when the network becomes more or less congested, the UE exceeds a quota, or the network policy changes (e.g., the user signs up for a new pricing plan). The UPF then performs traffic policing to enforce the bandwidth cap, along with packet scheduling to ensure a minimum bandwidth in conjunction with admission control in the control plane. \end{itemize} \sphinxAtStartPar The rest of this section describes two complementary strategies for implementing a UPF, one server\sphinxhyphen{}based and one switch\sphinxhyphen{}based. \subsection{5.4.1 Microservice Implementation} \label{\detokenize{core:microservice-implementation}} \sphinxAtStartPar A seemingly straightforward approach to supporting the set of Match/Action rules just described is to implement the UPF in software on a commodity server. Like any software\sphinxhyphen{}based router, the process would read a packet from an input port, classify the packet by matching it against a table of configured PDRs, execute the associated action(s), and then write the packet to an output port. Such a process could then be packaged as a Docker container, with one or more instances spun up on a Kubernetes cluster as workload dictates. This is mostly consistent with a microservice\sphinxhyphen{}based approach, with one important catch: the actions required to process each packet are stateful. \sphinxAtStartPar What we mean by this is that the UPF has two pieces of state that needs to be maintained on a per\sphinxhyphen{}UE / per\sphinxhyphen{}direction / per\sphinxhyphen{}class basis: (1) a finite state machine that transitions between \sphinxtitleref{forward}, \sphinxtitleref{tunnel}, \sphinxtitleref{buffer}, and \sphinxtitleref{notify}; and (2) a corresponding packet buffer when in \sphinxtitleref{buffer} state. This means that as the UPF scales up to handle more and more traffic—by adding a second, third, and fourth instance—packets still need to be directed to the original instance that knows the state for that particular flow. This breaks a fundamental assumption of a truly horizontally scalable service, in which traffic can be randomly directed to any instance in a way that balances the load. It also forces you to do packet classification before selecting which instance is the right one, which can potentially become a performance bottleneck, although it is possible to offload the classification stage to a SmartNIC/IPU. \subsection{5.4.2 P4 Implementation} \label{\detokenize{core:p4-implementation}} \sphinxAtStartPar Since the UPF is fundamentally an IP packet forwarding engine, it can also be implemented—at least in part—as a P4 program running on a programmable switch. Robert MacDavid and colleagues describe how that is done in SD\sphinxhyphen{}Core, which builds on the base packet forwarding machinery described in our companion SDN book. For the purposes of this section, the focus is on the four main challenges that are unique to implementing the UPF in P4. \phantomsection\label{\detokenize{core:reading-p4-upf}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar R. MacDavid, \sphinxstyleemphasis{et al.} \sphinxhref{https://www.cs.princeton.edu/~jrex/papers/up4-sosr21.pdf}{A P4\sphinxhyphen{}based 5G User Plane Function}. Symposium on SDN Research, September 2021. \sphinxAtStartPar \sphinxhref{https://sdn.systemsapproach.org}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. November 2021. \end{sphinxadmonition} \sphinxAtStartPar First, P4\sphinxhyphen{}programmable forwarding pipelines include an explicit “matching” mechanism built on \sphinxstyleemphasis{Ternary Content\sphinxhyphen{}Addressable Memory (TCAM)}. This memory supports fast table lookups for patterns that include wildcards, making it ideal for matching IP prefixes. In the case of the UPF, however, the most common PDRs correspond to exact matches of IP addresses (for downlink traffic to each UE) and GTP tunnel identifiers (for uplink traffic from each UE). More complex PDRs might include regular expressions for DNS names or require deep packet inspection. \sphinxAtStartPar Because TCAM capacity is limited, and the number of unique PDRs that need to be matched in both directions is potentially in the tens of thousands, it’s necessary to use the TCAM judiciously. One implementation strategy is to set up two parallel PDR tables: one using the relatively plentiful switch SRAM for common\sphinxhyphen{}case uplink rules that exactly matches on tunnel identifiers (which can be treated as table indices); and one using TCAM for common\sphinxhyphen{}case downlink rules that match the IP destination address. \sphinxAtStartPar Second, when a packet arrives from the Internet destined for an idle UE, the UPF buffers the packet and sends an alert to the 5G control plane, asking that the UE be awakened. Today’s P4\sphinxhyphen{}capable switches do not have large buffers or the ability to hold packets indefinitely, but a buffering microservice running on a server can be used to address this limitation. The microservice indefinitely holds any packets that it receives, and later releases them back to the switch when instructed to do so. The following elaborates on how this would work. \sphinxAtStartPar When the Mobile Core detects that a UE has gone idle (or is in the middle of a handover), it creates a FAR with the \sphinxtitleref{buffer} flag set, causing the on\sphinxhyphen{}switch P4 program to redirect packets to the buffering microservice. Packets are redirected without modifying their IP headers by placing them in a tunnel, using the same tunneling protocol that is used to send data to base stations. This allows the switch to treat the buffering microservice just like another base station. \sphinxAtStartPar When the first packet of a flow arrives at the buffering microservice, it sends an alert to the CP, which then (1) wakes up the UE, (2) modifies the corresponding FAR by unsetting the \sphinxtitleref{buffer} flag and setting the \sphinxtitleref{tunnel} flag, and once the UE is active, (3) instructs the buffering microservice to release all packets back to the switch. Packets arriving at the switch from the buffering microservice skip the portion of the UPF module they encountered before buffering, giving the illusion they are being buffered in the middle of the switch. That is, their processing resumes at the tunneling stage, where they are encapsulated and routed to the appropriate base station. \sphinxAtStartPar Third, QERs cannot be fully implemented in the switch because P4 does not include support for programming the packet scheduler. However, today’s P4 hardware does include fixed\sphinxhyphen{}function schedulers with configurable weights and priorities; these parameters are set using a runtime interface unrelated to P4. A viable approach, similar to the one MacDavid, Chen, and Rexford describe in their INFOCOM paper, is to map each QoS class specified in a QER onto one of the available queues, and assign a weight to that queue proportional to the fraction of the available bandwidth the class is to receive. As long as each class/queue is not over subscribed, individual UEs in the class will receive approximately the bit rate they have been promised. As an aside, since 3GPP under\sphinxhyphen{}specifies QoS guarantees (leaving the details to the implementation), such an approach is 3GPP\sphinxhyphen{}compliant. \phantomsection\label{\detokenize{core:reading-p4-qos}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar R. MacDavid, X. Chen, J. Rexford. \sphinxhref{https://www.cs.princeton.edu/~jrex/papers/infocom23.pdf}{Scalable Real\sphinxhyphen{}time Bandwidth Fairness in Switches}. IEEE INFOCOM, May 2023. \end{sphinxadmonition} \sphinxAtStartPar Finally, while the above description implies the Mobile Core’s CP talks directly to the P4 program on the switch, the implementation is not that straightforward. From the Core’s perspective, the SMF is responsible for sending/receiving control information to/from the UPF, but the P4 program implementing the UPF is controlled through an interface (known as P4Runtime or P4RT) that is auto\sphinxhyphen{}generated from the P4 program being controlled. MacDavid’s paper describes how this is done in more detail (and presumes a deep understanding of the P4 toolchain), but it can be summarized as follows. It is necessary to first write a “Model UPF” in P4, use that to program to generate the UPF\sphinxhyphen{}specific P4RT interface, and then write translators that (1) connect SMF to P4RT, and (2) connect P4RT to the underlying physical switches and servers. A high\sphinxhyphen{}level schematic of this software stack is shown in \hyperref[\detokenize{core:fig-p4-upf}]{Figure \ref{\detokenize{core:fig-p4-upf}}}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide23}.png} \caption{A model P4\sphinxhyphen{}based implementation of the UPF is used to generate the interface that is then used by the SMF running in the Mobile Core control plane to control the physical implementation of the UPF running on a combination of hardware switches and servers.}\label{\detokenize{core:id7}}\label{\detokenize{core:fig-p4-upf}}\end{figure} \sphinxAtStartPar Note that while this summary focuses on how the CP controls the UPF (the downward part of the schematic shown in \hyperref[\detokenize{core:fig-p4-upf}]{Figure \ref{\detokenize{core:fig-p4-upf}}}), the usage counters needed to generate URRs that flow upward to the CP are easy to support because the counters implemented in the switching hardware are identical to the counters in the Model UPF. When the Mobile Core requests counter values from the Model UPF, the backend translator polls the corresponding hardware switch counters and relays the response. \chapter{Chapter 6: Managed Cloud Service} \label{\detokenize{cloud:chapter-6-managed-cloud-service}}\label{\detokenize{cloud::doc}} \sphinxAtStartPar This chapter describes how to assemble all the pieces described in the previous chapters to provide 5G connectivity as a managed cloud service. Such a service might be deployed in enterprises, for example, in support of collection of operational data, video, robots, IoT devices, and so on—a set of use cases sometimes referred to as Industry 4.0. \sphinxAtStartPar The first step is to implement all the components using cloud native building blocks. We start by introducing those building blocks in Section 6.1. The second step is to introduce yet another component—a \sphinxstyleemphasis{Cloud Management Platform}—that is responsible for operationalizing 5G\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service. The rest of the sections describe how to build such a management system using open source tools. \sphinxAtStartPar Before getting into the details, it is important to remember that mobile cellular service (both voice and broadband) has been offered as a Telco service for 40 years. Treating it as a managed cloud service is a significant departure from that history, most notably in how the connectivity it provides is operated and managed. As a consequence, the Cloud Management Platform described in this chapter is significantly different from the legacy OSS/BSS mechanisms that have traditionally been the centerpiece of the Telco management machinery. The terminology is also different, but that only matters if you are trying to map Telco terminology onto cloud terminology (which we are not). We take up the “terminology mapping problem” in a companion book, and here focus instead on a from\sphinxhyphen{}scratch cloud\sphinxhyphen{}based design. \phantomsection\label{\detokenize{cloud:reading-ops}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar L. Peterson, A. Bavier, S. Baker, Z. Williams, and B. Davie. \sphinxhref{https://ops.systemsapproach.org/lifecycle.html}{Edge Cloud Operations: A Systems Approach}. June 2022. \end{sphinxadmonition} \section{6.1 Building Blocks} \label{\detokenize{cloud:building-blocks}} \sphinxAtStartPar The implementation strategy starts with commodity hardware and open source software. These building blocks will be familiar to anyone who has built a cloud native application, but they deserve to be explicitly named in a discussion of mobile cellular networks, which have historically been built using closed, proprietary hardware devices. \sphinxAtStartPar The hardware building blocks include bare\sphinxhyphen{}metal servers and switches, which might include ARM or x86 processor chips and Tomahawk or Tofino switching chips, respectively. A physical cloud cluster is then constructed with the hardware building blocks arranged as shown in \hyperref[\detokenize{cloud:fig-hw}]{Figure \ref{\detokenize{cloud:fig-hw}}}: one or more racks of servers connected by a leaf\sphinxhyphen{}spine switching fabric. We show the servers above the switching fabric to emphasize that software running on the servers controls the switches (as we will see in the next section). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide41}.png} \caption{Example building block components used to construct an edge cloud, including commodity servers and switches, interconnected by a leaf\sphinxhyphen{}spine switching fabric.}\label{\detokenize{cloud:id1}}\label{\detokenize{cloud:fig-hw}}\end{figure} \sphinxAtStartPar The software building blocks start with the following open source components: \begin{enumerate} \sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}% \item {} \sphinxAtStartPar Docker containers package software functionality. \item {} \sphinxAtStartPar Kubernetes instantiates and interconnects a set of containers. \item {} \sphinxAtStartPar Helm specifies how collections of related containers are interconnected to build microservice\sphinxhyphen{}based applications. \item {} \sphinxAtStartPar Fleet specifies how a set of Kubernetes applications are to be deployed on the available infrastructure. \item {} \sphinxAtStartPar Terraform provisions a set of one or more Kubernetes clusters, configuring them to host microservice applications. \end{enumerate} \sphinxAtStartPar Docker is a container runtime that leverages OS isolation APIs to instantiate and run multiple containers, each of which is an instance defined by a Docker image. Docker images are most frequently built using a Dockerfile, which uses a layering approach that allows sharing and building customized images on top of base images. A final image for a particular task incorporates all dependencies required by the software that is to run in the container, resulting in a container image that is portable across servers, depending only on the kernel and Docker runtime. We also assume one or more image artifact repositories of Docker containers that we will want to deploy in our cloud, of which \sphinxurl{https://hub.docker.com/} is the best known example. \phantomsection\label{\detokenize{cloud:reading-docker}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://www.docker.com/101-tutorial}{Docker Tutorial}. \end{sphinxadmonition} \sphinxAtStartPar Kubernetes is a container orchestration system. It provides a programmatic interface for scaling container instances up and down, allocating server resources to them, setting up virtual networks to interconnect those instances, and opening service ports that external clients can use to access those instances. Behind the scenes, Kubernetes monitors the liveness of those containers, and automatically restarts any that have failed. In other words, if you instruct Kubernetes to spin up three instances of microservice X, Kubernetes will do its best to keep three instances of the container that implements X running at all times. \phantomsection\label{\detokenize{cloud:reading-k8s}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://kubernetes.io/docs/tutorials/kubernetes-basics/}{Kubernetes Tutorial}. \end{sphinxadmonition} \sphinxAtStartPar Helm is a configuration manager that runs on top of Kubernetes. It issues calls against the Kubernetes API according to a developer\sphinxhyphen{}provided specification, known as a \sphinxstyleemphasis{Helm Chart}. It is now common practice for cloud applications built from a set of microservices to publish a Helm chart that defines how the application is to be deployed on a Kubernetes cluster. See \sphinxurl{https://artifacthub.io/} for a collection of publicly available Helm Charts. \phantomsection\label{\detokenize{cloud:reading-helm}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://helm.sh/docs/intro/quickstart/}{Helm Tutorial}. \end{sphinxadmonition} \sphinxAtStartPar Fleet, an application deployment manager, is responsible for installing a \sphinxstyleemphasis{Bundle} of Helm Charts on one or more target clusters. If we were trying to deploy a single Chart on just one Kubernetes cluster, then Helm would be sufficient. The value of Fleet is that it scales up that process, helping us manage the deployment of multiple charts across multiple clusters. Moreover, Fleet does this using an approach known as \sphinxstyleemphasis{Configuration\sphinxhyphen{}as\sphinxhyphen{}Code}, where the desired configuration is checked into a repo, just like any other software. Checking a new or updated Bundle into a repo triggers the deployment of the corresponding applications. \phantomsection\label{\detokenize{cloud:reading-fleet}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://fleet.rancher.io/}{Fleet: GitOps at Scale}. \end{sphinxadmonition} \sphinxAtStartPar Terraform is an infrastructure manager that, in our scenario, provisions one or more Kubernetes clusters, preparing them to host a collection of Helm\sphinxhyphen{}specified applications. It does this using an approach known as \sphinxstyleemphasis{Infrastructure\sphinxhyphen{}as\sphinxhyphen{}Code}, which documents exactly how the infrastructure is to be configured in a declarative format that can be (a) checked into a repo, (b) version\sphinxhyphen{}controlled, and (c) executed just like any piece of software. Terraform assumes an underlying provisioning API, with Microsoft’s Azure Kubernetes Service (AKS), AWS’s Amazon Elastic Kubernetes Service (EKS), Google’s Google Kubernetes Engine (GKE) and Rancher’s Rancher Kubernetes Engine (RKE) being widely available examples. \phantomsection\label{\detokenize{cloud:reading-terraform}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://learn.hashicorp.com/terraform}{Terraform Tutorials}. \end{sphinxadmonition} \sphinxAtStartPar The inter\sphinxhyphen{}related roles of Helm, Fleet, and Terraform can be confusing, in part because there is overlap in what each tries to do. One distinction is that Helm Charts are typically specified by \sphinxstyleemphasis{developers} as a way of specifying how an application is constructed from a set of microservices, whereas Fleet and Terraform give \sphinxstyleemphasis{operators} an opportunity to specify details of their particular deployment scenarios. A second distinction is that Helm and Fleet help manage the \sphinxstyleemphasis{applications running on} one or more Kubernetes clusters, whereas Terraform is used to set up and configure the \sphinxstyleemphasis{underlying Kubernetes clusters} in the first place. Again, there is overlap in the capabilities of these respective tools, but these two distinctions characterize how they are used in Aether. The more general takeaway is that cloud management has to accommodate both developers and operators, and to clearly delineate between applications and platforms. \section{6.2 Example Deployment} \label{\detokenize{cloud:example-deployment}} \sphinxAtStartPar Using these building blocks, it is possible to construct a wide range of deployment scenarios for a managed 5G service. For illustrative purposes, we use a particular deployment based on the Aether edge cloud introduced in Chapter 2. Aether is an operational edge cloud that has been deployed to multiple sites, and most importantly for our purposes, includes an API that edge apps can use to customize 5G connectivity to better meet their objectives. \subsection{6.2.1 Edge Cloud} \label{\detokenize{cloud:edge-cloud}} \sphinxAtStartPar An Aether edge deployment, called ACE (Aether Connected Edge), is a Kubernetes\sphinxhyphen{}based cluster. It consists of one or more server racks interconnected by a leaf\sphinxhyphen{}spine switching fabric, with an SDN control plane (denoted SD\sphinxhyphen{}Fabric) managing the fabric. We briefly saw SD\sphinxhyphen{}Fabric in Chapter 5 as an implementation option for the Mobile Core’s User Plane Function (UPF), but for an in\sphinxhyphen{}depth description of SD\sphinxhyphen{}Fabric, we refer you to a companion book. \phantomsection\label{\detokenize{cloud:reading-sdn}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar L. Peterson, C. Cascone, B. O’Connor, T. Vachuska, and B. Davie. \sphinxhref{https://sdn.systemsapproach.org}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. November 2021. \end{sphinxadmonition} \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=350\sphinxpxdimen]{{Slide51}.png} \caption{Aether Connected Edge (ACE) = The cloud platform (Kubernetes and SD\sphinxhyphen{}Fabric) plus the 5G connectivity service (RAN and User Plane of Mobile Core). Dotted lines (e.g., between SD\sphinxhyphen{}RAN and the individual base stations, and between the Network OS and the individual switches) represent control relationships (e.g., SD\sphinxhyphen{}RAN controls the small cells and SD\sphinxhyphen{}Fabric controls the switches).}\label{\detokenize{cloud:id2}}\label{\detokenize{cloud:fig-ace}}\end{figure} \sphinxAtStartPar As shown in \hyperref[\detokenize{cloud:fig-ace}]{Figure \ref{\detokenize{cloud:fig-ace}}}, ACE hosts two additional microservice\sphinxhyphen{}based subsystems on top of this platform; they collectively implement \sphinxstyleemphasis{5G\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service}. The first subsystem, SD\sphinxhyphen{}RAN, is the SDN\sphinxhyphen{}based implementation of the Radio Access Network described in Chapter 4. It controls the small cell base stations deployed throughout the enterprise. The second subsystem, SD\sphinxhyphen{}Core, is an SDN\sphinxhyphen{}based implementation of the User Plane half of the Mobile Core described in Chapter 5. It is responsible for forwarding traffic between the RAN and the Internet. The SD\sphinxhyphen{}Core Control Plane (CP) runs off\sphinxhyphen{}site, and is not shown in \hyperref[\detokenize{cloud:fig-ace}]{Figure \ref{\detokenize{cloud:fig-ace}}}. Both subsystems (as well as the SD\sphinxhyphen{}Fabric), are deployed as a set of microservices, just as any other cloud native workload. \sphinxAtStartPar Once an edge cluster is running in this configuration, it is ready to host a collection of cloud\sphinxhyphen{}native edge applications (not shown in \hyperref[\detokenize{cloud:fig-ace}]{Figure \ref{\detokenize{cloud:fig-ace}}}). What’s unique to our example configuration is its ability to connect such applications to mobile devices throughout the enterprise using the 5G Connectivity Service implemented by SD\sphinxhyphen{}RAN and SD\sphinxhyphen{}Core, without the resulting network traffic ever leaving the enterprise; a scenario known as \sphinxstyleemphasis{local breakout}. Moreover, this service is offered as a managed service, with enterprise system administrators able to use a programmatic API (and associated GUI portal) to control that service; that is, authorize devices, restrict access, set QoS profiles for different devices and applications, and so on. \subsection{6.2.2 Hybrid Cloud} \label{\detokenize{cloud:hybrid-cloud}} \sphinxAtStartPar While it is possible to instantiate a single ACE cluster in just one site, Aether is designed to support multiple edge deployments, all of which are managed from the central cloud. Such a hybrid cloud scenario is depicted in \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}}, which shows two subsystems running in the central cloud: (1) one or more instantiations of the Mobile Core Control Plane (CP), and (2) the Aether Management Platform (AMP). \sphinxAtStartPar Each SD\sphinxhyphen{}Core CP controls one or more SD\sphinxhyphen{}Core UPFs. Exactly how CP instances (running centrally) are paired with UPF instances (running at the edges) is a runtime decision, and depends on the degree of isolation the enterprise sites require. AMP is Aether’s realization of a Cloud Management Platform; it is responsible for managing all the centralized and edge subsystems (as introduced in the next section). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide61}.png} \caption{Aether runs in a hybrid cloud configuration, with Control Plane of Mobile Core and the Aether Management Platform (AMP) running in the Central Cloud.}\label{\detokenize{cloud:id3}}\label{\detokenize{cloud:fig-aether}}\end{figure} \sphinxAtStartPar There is an important aspect of this hybrid cloud that is not obvious from \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}}, which is that the “hybrid cloud” we keep referring to is best described as a set of Kubernetes clusters, rather than a set of physical clusters. This is because, while each ACE site usually corresponds to a physical cluster built out of bare\sphinxhyphen{}metal components, each of the SD\sphinxhyphen{}Core CP subsystems shown in \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}} is actually deployed in a logical Kubernetes cluster on a commodity cloud. The same is true for AMP. Aether’s centralized components are able to run in Google Cloud Platform, Microsoft Azure, and Amazon’s AWS. They also run as an emulated cluster implemented by a system like KIND (Kubernetes in Docker), making it possible for developers to run these components on their laptops. \begin{sphinxShadowBox} \sphinxstylesidebartitle{Near\sphinxhyphen{}Edge vs Far\sphinxhyphen{}Edge} \sphinxAtStartPar \sphinxstyleemphasis{We use enterprises as the exemplar edge deployment in this book, without prescribing a role for traditional MNOs. When traditional MNOs are involved, it is not uncommon for them to make a distinction between the “near\sphinxhyphen{}edge” and the “far\sphinxhyphen{}edge”, where the far\sphinxhyphen{}edge corresponds to the enterprise and the near\sphinxhyphen{}edge corresponds to their traditional aggregation points (or Central Offices), as described in Section 1.2. In such a scenario, it is typically the case that the RU and DU are located at the far\sphinxhyphen{}edge (on\sphinxhyphen{}prem), while the CU—along with both the Control and User Planes of the Mobile Core—run in the near\sphinxhyphen{}edge. Such a configuration does not support local breakout, since all traffic must travel to the near\sphinxhyphen{}edge before being routed to the edge app (which might be running back in the enterprise).} \sphinxAtStartPar \sphinxstyleemphasis{In contrast, the deployment described in this Chapter has everything except the Mobile Core Control Plane (CP) running on\sphinxhyphen{}prem. Moreover, because there is no traditional MNO involved, there is no near\sphinxhyphen{}edge to speak of, with the Core CP instead running in a central cloud. For example, this section describes a deployment with SD\sphinxhyphen{}Core (CP) running in the Google Cloud. It is the case, however, that the SD\sphinxhyphen{}Core (CP) can optionally run on\sphinxhyphen{}prem if a fully local configuration is preferred. Where each component runs is a configuration option.} \end{sphinxShadowBox} \subsection{6.2.3 Stakeholders} \label{\detokenize{cloud:stakeholders}} \sphinxAtStartPar With the understanding that our target environment is a collection of Kubernetes clusters—some running on bare\sphinxhyphen{}metal hardware at edge sites and some running in central datacenters—there is an orthogonal issue of how decision\sphinxhyphen{}making responsibility for those clusters is shared among multiple stakeholders. Identifying the relevant stakeholders is an important prerequisite for establishing a cloud service, and while the example we use may not be suitable for all situations, it does illustrate the design implications. \sphinxAtStartPar For Aether, we care about two primary stakeholders: (1) the \sphinxstyleemphasis{cloud operators} who manage the hybrid cloud as a whole, and (2) the \sphinxstyleemphasis{enterprise users} who decide on a per\sphinxhyphen{}site basis how to take advantage of the local cloud resources (e.g., what edge applications to run and how to slice connectivity resources among those apps). We sometimes call the latter “enterprise admins” to distinguish them from “end\sphinxhyphen{}users” who might want to manage their own personal devices. \sphinxAtStartPar Aether is multi\sphinxhyphen{}tenant in the sense that it authenticates and isolates these stakeholders, allowing each to access only those objects they are responsible for. This makes the approach agnostic as to whether all the edge sites belong to a single organization (with that organization also responsible for operating the cloud), or alternatively, there being a separate organization that offers a managed service to a set of distinct enterprises (each of which spans one or more sites). \sphinxAtStartPar There is a third stakeholder of note—third\sphinxhyphen{}party service providers—which points to the larger issue of how we deploy and manage the edge applications that take advantage of 5G\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service. The approach Aether adopts is to expect service providers to make their applications available either as source code (which works for open source or in\sphinxhyphen{}house apps), or as standard cloud native artifacts (e.g., Docker images and Helm charts). Either format can be fed into the Lifecycle Management pipeline described in Section 6.3.2. The alternative would be for edge service providers to share operational responsibility for the edge cloud with the cloud operator, which is possible if the infrastructure running at the edge is either multi\sphinxhyphen{}tenant or a multi\sphinxhyphen{}cloud. \subsection{6.2.4 Alternative Configurations} \label{\detokenize{cloud:alternative-configurations}} \sphinxAtStartPar The deployment just described is Aether in its full glory. Simpler configurations are also possible, which makes sense in less demanding scenarios. Examples include: \begin{itemize} \item {} \sphinxAtStartPar Small edge clusters can be built with only a single switch (or two switches for resiliency), with or without SDN\sphinxhyphen{}based control. In the limit, an Aether edge can run on a single server. \item {} \sphinxAtStartPar It is possible to substitute legacy small cells for O\sphinxhyphen{}RAN compliant small cells and the SD\sphinxhyphen{}RAN solution that includes a near RT\sphinxhyphen{}RIC and associated xApps. \item {} \sphinxAtStartPar It is possible co\sphinxhyphen{}locate both AMP and the SD\sphinxhyphen{}Core on the edge cluster, resulting in a complete Aether deployment that is self\sphinxhyphen{}contained in a single site. \end{itemize} \sphinxAtStartPar These are all straightforward configuration options. A very different approach is to start with an edge cluster that is managed by one of the hyperscalers, rather than have Aether provision Kubernetes on bare\sphinxhyphen{}metal. Google’s Anthos, Microsoft’s Azure Arc, and Amazon’s ECS\sphinxhyphen{}Anywhere are examples of such edge cloud products. In such a scenario, AMP still manages the SD\sphinxhyphen{}Core and SD\sphinxhyphen{}RAN applications running on top of Kubernetes, but not the underlying platform (which may or may not include an SDN\sphinxhyphen{}based switching fabric). \sphinxAtStartPar Another variable in how 5G can be deployed at the edge is related to who owns the underlying cloud infrastructure. Instead of a cloud provider, an enterprise, or a traditional MNO owning the hardware, there are situations where a third\sphinxhyphen{}party, often called a \sphinxstyleemphasis{neutral host}, owns and operates the hardware (along with the real estate it sits in), and then rents access to these resources to multiple 5G providers. Each mobile service provider is then a tenant of that shared infrastructure. \sphinxAtStartPar This kind of arrangement has existed for years, albeit with conventional RAN devices, but shifting to a cloud\sphinxhyphen{}based design makes it possible for neutral hosts to lease access to \sphinxstyleemphasis{virtualized} edge resources to their tenants. In principle, the only difference between this scenario and today’s multi\sphinxhyphen{}tenant clouds is that such providers would offer edge resources—located in cell towers, apartment buildings, and dense urban centers—instead of datacenter resources. The business arrangements would also have to be different from Private 5G, but the technical design outlined in this book still applies. \section{6.3 Cloud Management Platform} \label{\detokenize{cloud:cloud-management-platform}} \sphinxAtStartPar Operationalizing the hardware and software components described in the previous two sections is the essence of what it means to offer 5G as a \sphinxstyleemphasis{managed service}. This responsibility falls to the Cloud Management Platform, which in Aether corresponds to the centralized AMP component shown in \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}}. AMP manages both the distributed set of ACE clusters and one or more SD\sphinxhyphen{}Core CP clusters running in the central cloud. \sphinxAtStartPar The following uses AMP to illustrate how to deliver 5G\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service, but the approach generalizes because AMP is based on widely\sphinxhyphen{}used open source tools. For more details about all the subsystems involved in operationalizing an edge cloud, we refer you to the companion book mentioned in the introduction to this chapter. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide71}.png} \caption{The four subsystems that comprise AMP: Resource Provisioning, Lifecycle Management, Service Orchestrator, and Monitoring \& Telemetry.}\label{\detokenize{cloud:id4}}\label{\detokenize{cloud:fig-amp}}\end{figure} \sphinxAtStartPar At a high level, AMP is organized around the four subsystems shown in \hyperref[\detokenize{cloud:fig-amp}]{Figure \ref{\detokenize{cloud:fig-amp}}}: \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Resource Provisioning} is responsible for initializing resources (e.g., servers, switches) that add, replace, or upgrade capacity. It configures and bootstraps both physical and virtual resources, bringing them up to a state so Lifecycle Management can take over and manage the software running on those resources. \item {} \sphinxAtStartPar \sphinxstylestrong{Lifecycle Management} is responsible for continuous integration and deployment of the software components that collectively implement 5G\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service. It adopts the GitOps practice of \sphinxstyleemphasis{Configuration\sphinxhyphen{}as\sphinxhyphen{}Code}, using Helm Charts, Terraform Templates, and Fleet Bundles to specify how functionality is to be deployed and configured. \item {} \sphinxAtStartPar \sphinxstylestrong{Service Orchestration} provides a means to manage services once they are operational. It defines an API that hides the implementation details of the underlying microservices, and is used to manage the provided 5G connectivity service. \item {} \sphinxAtStartPar \sphinxstylestrong{Monitoring \& Telemetry} is responsible for collecting, archiving, evaluating, and analyzing operational data generated by the underlying components. It makes it possible to diagnose and respond to failures, tune performance, do root cause analysis, perform security audits, and understand when it is necessary to provision additional capacity. \end{itemize} \sphinxAtStartPar AMP implements all four subsystems, but an alternative perspective that characterizes the management platform as having \sphinxstyleemphasis{online} and \sphinxstyleemphasis{offline} components is also instructive. Such a two dimensional schematic is shown in \hyperref[\detokenize{cloud:fig-2d}]{Figure \ref{\detokenize{cloud:fig-2d}}}. Lifecycle Management (coupled with Resource Provisioning) runs offline, sitting adjacent to the hybrid cloud. Operators and Developers provision and change the system by checking code (including configuration specs) into a repo, which in turn triggers an upgrade of the running system. Service Orchestration (coupled with Monitoring and Telemetry) runs online, layered on top of the hybrid cloud being managed. It defines an API that can be used to read and write parameters of the running system, which serves as a foundation for building closed\sphinxhyphen{}loop control. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide111}.png} \caption{Alternative representation of the management platform, highlighting the offline and online aspects of cloud management.}\label{\detokenize{cloud:id5}}\label{\detokenize{cloud:fig-2d}}\end{figure} \sphinxAtStartPar The offline and online aspects of cloud management are related in the sense that the offline component is also responsible for lifecycle\sphinxhyphen{}managing the online component. This is because the latter is deployed as a collection of Kubernetes applications, just like SD\sphinxhyphen{}Core and SD\sphinxhyphen{}RAN. Version management is a key aspect of this relationship since the runtime API to the 5G connectivity service has to stay in sync with the underlying implementation of the constituent subsystems. How Aether realizes version control is described in more detail in the companion Edge Cloud Operations book. \subsection{6.3.1 Resource Provisioning} \label{\detokenize{cloud:resource-provisioning}} \sphinxAtStartPar Resource Provisioning is the process of bringing virtual and physical resources online. For physical resources, it has both a hands\sphinxhyphen{}on component (racking and connecting devices) and a bootstrap component (configuring how the resources boot into a “ready” state). When utilizing virtual resources (e.g., VMs instantiated on a commercial cloud) the “rack and connect” step is carried out by a sequence of API calls rather than a hands\sphinxhyphen{}on technician. \sphinxAtStartPar Because we want to automate the sequence of calls needed to activate virtual infrastructure, we adopt an approach known as \sphinxstyleemphasis{Infrastructure\sphinxhyphen{}as\sphinxhyphen{}Code}. This is where Terraform comes into play. The general idea is to document, in a declarative format that can be “executed”, exactly what our infrastructure is to look like. The code defines how the infrastructure is to be configured. \sphinxAtStartPar When a cloud is built from a combination of virtual and physical resources, as is the case for a hybrid cloud like Aether, we need a seamless way to accommodate both. To this end, our approach is to first overlay a \sphinxstyleemphasis{logical structure} on top of hardware resources, making them roughly equivalent to the virtual resources we get from a commercial cloud provider. This results in a hybrid scenario similar to the one shown in \hyperref[\detokenize{cloud:fig-infra}]{Figure \ref{\detokenize{cloud:fig-infra}}}. One way to think about this is that the task of booting hardware into the “ready” state involves installing and configuring several subsystems that collectively form the cloud platform. It is this platform that Terraform interacts with, indirectly, through a cloud provisioning API. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide121}.png} \caption{Resource Provisioning in a hybrid cloud that includes both physical and virtual resources.}\label{\detokenize{cloud:id6}}\label{\detokenize{cloud:fig-infra}}\end{figure} \subsection{6.3.2 Lifecycle Management} \label{\detokenize{cloud:lifecycle-management}} \sphinxAtStartPar Lifecycle Management is concerned with updating and evolving a running system over time. \hyperref[\detokenize{cloud:fig-cicd}]{Figure \ref{\detokenize{cloud:fig-cicd}}} gives an overview of the pipeline/toolchain that make up the two halves of Lifecycle Management—Continuous Integration (CI) and Continuous Deployment (CD). The key thing to focus on is the Image and Config Repos in the middle. They represent the “interface” between the two halves: CI produces Docker Images and Helm Charts, storing them in the respective Repositories, while CD consumes Docker Images and Helm Charts, pulling them from the respective Repositories. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide81}.png} \caption{Overview of the CI/CD pipeline.}\label{\detokenize{cloud:id7}}\label{\detokenize{cloud:fig-cicd}}\end{figure} \sphinxAtStartPar The Config Repo also contains declarative specifications of the infrastructure artifacts (specifically, Terraform templates and Fleet Bundles). These files are input to Lifecycle Management, which implies that Terraform and Fleet gets invoked as part of CI/CD whenever these files change. In other words, CI/CD keeps both the software\sphinxhyphen{}related components in the underlying cloud platform and the microservice workloads that run on top of that platform up to date. \begin{sphinxShadowBox} \sphinxstylesidebartitle{Continuous Delivery vs Deployment} \sphinxAtStartPar \sphinxstyleemphasis{You will also hear CD refer to “Continuous Delivery” instead of “Continuous Deployment”, but we are interested in the complete end\sphinxhyphen{}to\sphinxhyphen{}end process, so CD will always imply the latter in this book. But keep in mind that “continuous” does not necessarily mean “instantaneous”; there can be a variety of gating functions injected into the CI/CD pipeline to control when and how upgrades get rolled out. The important point is that all the stages in the pipeline are automated.} \sphinxAtStartPar \sphinxstyleemphasis{So what exactly does “Continuous Delivery” mean? Arguably, it’s redundant when coupled with “Continuous Integration” since the set of artifacts being produced by the CI half of the pipeline (e.g., Docker images) is precisely what’s being delivered. There is no “next step” unless you also deploy those artifacts. It’s hair\sphinxhyphen{}splitting, but some would argue CI is limited to testing new code and Continuous Delivery corresponds to the final “publish the artifact” step. For our purposes, we lump “publish the artifact” into the CI half of the pipeline.} \end{sphinxShadowBox} \sphinxAtStartPar There are three takeaways from this overview. The first is that by having well\sphinxhyphen{}defined artifacts passed between CI and CD (and between operators responsible for resource provisioning and CD), the subsystems are loosely coupled, and able to perform their respective tasks independently. The second is that all authoritative state needed to successfully build and deploy the system is contained within the pipeline, specifically, as declarative specifications in the Config Repo. This is the cornerstone of \sphinxstyleemphasis{Configuration\sphinxhyphen{}as\sphinxhyphen{}Code} (also known as \sphinxstyleemphasis{GitOps}), the cloud native approach to CI/CD. The third is that there is an opportunity for operators to apply discretion to the pipeline, as denoted by the \sphinxstyleemphasis{“Deployment Gate”} in the Figure, controlling what features get deployed when. \sphinxAtStartPar The third repository shown in \hyperref[\detokenize{cloud:fig-cicd}]{Figure \ref{\detokenize{cloud:fig-cicd}}} is the Code Repo (on the far left). Developers continually check new features and bug fixes into this repo, which triggers the CI/CD pipeline. A set of tests and code reviews are run against these check\sphinxhyphen{}ins, with the output of those tests/reviews reported back to developers, who modify their patch sets accordingly. (These develop\sphinxhyphen{}and\sphinxhyphen{}test feedback loops are implied by the dotted lines in \hyperref[\detokenize{cloud:fig-cicd}]{Figure \ref{\detokenize{cloud:fig-cicd}}}.) \sphinxAtStartPar The far right of \hyperref[\detokenize{cloud:fig-cicd}]{Figure \ref{\detokenize{cloud:fig-cicd}}} shows the set of deployment targets, with \sphinxstyleemphasis{Staging} and \sphinxstyleemphasis{Production} called out as two illustrative examples. The idea is that a new version of the software is deployed first to a set of Staging clusters, where it is subjected to realistic workloads for a period of time, and then rolled out to the Production clusters once the Staging deployments give us confidence that the upgrade is reliable. \sphinxAtStartPar Finally, two of the CI stages shown in \hyperref[\detokenize{cloud:fig-cicd}]{Figure \ref{\detokenize{cloud:fig-cicd}}} identify a \sphinxstyleemphasis{Testing} component. One is a set of component\sphinxhyphen{}level tests that are run against each patch set checked into the Code Repo. These tests gate integration; fully merging a patch into the Code Repo requires first passing this preliminary round of tests. Once merged, the pipeline runs a build across all the components, and a second round of testing happens on a \sphinxstyleemphasis{Quality Assurance (QA)} cluster. Passing these tests gate deployment, but as just noted, testing also happens in the Staging clusters as part of the CD end of the pipeline. \subsection{6.3.3 Service Orchestration} \label{\detokenize{cloud:service-orchestration}} \sphinxAtStartPar Service Orchestration is responsible for managing the Kubernetes workloads once they are up and running, which in our case means providing a programmatic API that can be used by various stakeholders to manage the 5G connectivity service. As shown in \hyperref[\detokenize{cloud:fig-control}]{Figure \ref{\detokenize{cloud:fig-control}}}, the Service Orchestrator hides the implementation details of 5G connectivity, which spans four different components and multiple clouds. It does this by providing a coherent service interface for users, enabling them to authorize devices and set QoS parameters on an end\sphinxhyphen{}to\sphinxhyphen{}end basis. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide91}.png} \caption{Example use case that requires ongoing runtime control.}\label{\detokenize{cloud:id8}}\label{\detokenize{cloud:fig-control}}\end{figure} \sphinxAtStartPar In other words, the Service Orchestrator defines an abstraction layer on top of a collection of backend components, effectively turning them into an externally visible (and controllable) cloud service. In some situations a single backend component might implement the entirety of a service, but in the case of 5G, which is constructed from a collection of disaggregated components, Service Orchestration is where we define an API that logically integrates those components into a unified and coherent whole. It is also an opportunity to “raise the level of abstraction” for the underlying subsystems, hiding unnecessary implementation details. \sphinxAtStartPar We describe this connectivity interface in Section 6.4. For now, our focus is on the main issues Service Orchestration must address in order to offer such an API. At a high level, it must: \begin{enumerate} \sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}% \item {} \sphinxAtStartPar Authenticate the principal wanting to perform the operation. \item {} \sphinxAtStartPar Determine if that principal has sufficient privilege to carry out the operation. \item {} \sphinxAtStartPar Push the new parameter setting(s) to one or more backend components. \item {} \sphinxAtStartPar Record the specified parameter setting(s), so the new value(s) persist. \end{enumerate} \sphinxAtStartPar Central to this role is the requirement that Service Orchestration be able to represent a set of abstract objects, which is to say, it implements a \sphinxstyleemphasis{data model}. The API is then generated from this data model, and persistent state associated with instances of the models is stored in a key\sphinxhyphen{}value store. Aether uses YANG to specify the models, in part because it is a rich language for data modeling, but also because there is a robust collection of YANG\sphinxhyphen{}based tools that we can build upon. \phantomsection\label{\detokenize{cloud:reading-yang}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://datatracker.ietf.org/doc/html/rfc6020}{YANG \sphinxhyphen{} A Data Modeling Language for the Network Configuration Protocol}. RFC 6020. October 2010. \end{sphinxadmonition} \sphinxAtStartPar Finally, changes to the model\sphinxhyphen{}defined parameters must be propagated to the backend components, and in practice there is no established API for doing this. Aether assumes gNMI as its southbound interface to communicate configuration changes to the software services, where an Adapter (not shown in the figure) has to be written for any services that do not support gNMI natively. \subsection{6.3.4 Monitoring and Telemetry} \label{\detokenize{cloud:monitoring-and-telemetry}} \sphinxAtStartPar Collecting telemetry data for a running system is an essential function of the management platform. It enables operators to monitor system behavior, evaluate performance, make informed provisioning decisions, respond to failures, identify attacks, and diagnose problems. There are three types of telemetry data—\sphinxstyleemphasis{metrics}, \sphinxstyleemphasis{logs}, and \sphinxstyleemphasis{traces}—along with open source software stacks available to help collect, store, and act upon each of them. \sphinxAtStartPar Metrics are quantitative data about a system. These include common performance metrics such as link bandwidth, CPU utilization, and memory usage, but also binary results corresponding to “up” and “down”, as well as other state variables that can be encoded numerically. These values are produced and collected periodically (e.g., every few seconds), either by reading a counter, or by executing a runtime test that returns a value. These metrics can be associated with physical resources such as servers and switches, virtual resources such as VMs and containers, or high\sphinxhyphen{}level abstractions such as the \sphinxstyleemphasis{Connectivity Service} described in the next section. Given these many possible sources of data, the job of the metrics monitoring stack is to collect, archive, visualize, and optionally analyze this data. Prometheus is a popular open source tool for storing and querying metrics. \phantomsection\label{\detokenize{cloud:reading-monitor}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://prometheus.io/docs/introduction/overview/}{Prometheus}. \end{sphinxadmonition} \sphinxAtStartPar Logs are the qualitative data that is generated whenever a noteworthy event occurs. This information can be used to identify problematic operating conditions (i.e., it may trigger an alert), but more commonly, it is used to troubleshoot problems after they have been detected. Various system components—all the way from the low\sphinxhyphen{}level OS kernel to high\sphinxhyphen{}level cloud services—write messages that adhere to a well\sphinxhyphen{}defined format to the log. These messages include a timestamp, which makes it possible for the logging stack to parse and correlate messages from different components. ElasticSearch is a widely\sphinxhyphen{}used tool for storing and analyzing log messages. \phantomsection\label{\detokenize{cloud:reading-logging}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://www.elastic.co/elasticsearch/}{ElasticSearch}. \end{sphinxadmonition} \sphinxAtStartPar Traces are a record of causal relationships (e.g., Service A calls Service B) resulting from user\sphinxhyphen{}initiated transactions or jobs. They are related to logs, but provide more specialized information about the context in which different events happen. Tracing is well understood in a single program, where an execution trace is commonly recorded as an in\sphinxhyphen{}memory call stack, but traces are inherently distributed across a graph of network\sphinxhyphen{}connected microservices in a cloud setting. This makes the problem challenging, but also critically important because it is often the case that the only way to understand time\sphinxhyphen{}dependent phenomena—such as why a particular resource is overloaded—is to understand how multiple independent workflows interact with each other. Jaeger is a popular open source tool used for tracing. \phantomsection\label{\detokenize{cloud:reading-tracing}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar \sphinxhref{https://www.jaegertracing.io/}{Jaeger: End\sphinxhyphen{}to\sphinxhyphen{}End Distributed Tracing}. \end{sphinxadmonition} \sphinxAtStartPar Finally, note that our framing of monitoring and telemetry as part of the online aspect of management is somewhat simplistic. Certainly telemetry data is collected from online processes embedded in a running system, and such data can be coupled with online control operations to realize closed\sphinxhyphen{}loop control, but it is also the case that some telemetry data is evaluated offline. This is true for logs and traces used to diagnose problems, and for performance data used to make provisioning decisions, both of which can lead to code changes that feed back into the next iteration of lifecycle management. \section{6.4 Connectivity API} \label{\detokenize{cloud:connectivity-api}} \sphinxAtStartPar The visible aspect of a 5G service is the programmatic interface it provides to users, giving them the ability to control and customize the underlying connectivity service. This API is implemented by the Service Orchestrator outlined in the previous section, but what we really care about is the interface itself. Using Aether as a concrete example, this section describes such an API. \sphinxAtStartPar Like many cloud services, the API for 5G\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service is RESTful. This means it supports REST’s GET, POST, PATCH, and DELETE operations on a set of resources (objects): \begin{itemize} \item {} \sphinxAtStartPar GET: Retrieve an object. \item {} \sphinxAtStartPar POST: Create an object. \item {} \sphinxAtStartPar PUT, PATCH: Modify an existing object. \item {} \sphinxAtStartPar DELETE: Delete an object. \end{itemize} \sphinxAtStartPar Each object, in turn, is typically defined by a data model. In Aether this model is specified in YANG, but rather than dive into the particulars of YANG, this section describes the models informally by describing the relevant fields. \sphinxAtStartPar Every object contains an \sphinxtitleref{id} field that is used to uniquely identify the object. Some objects contain references to other objects. For example, many objects contain references to the \sphinxtitleref{Enterprise} object, which allows them to be associated with a particular enterprise. That is, references are constructed using the \sphinxtitleref{id} field of the referenced object. \sphinxAtStartPar In addition to the \sphinxtitleref{id} field, several other fields are also common to all models. These include: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{description}: A human\sphinxhyphen{}readable description, used to store additional context about the object. \item {} \sphinxAtStartPar \sphinxtitleref{display\sphinxhyphen{}name}: A human\sphinxhyphen{}readable name that is shown in the GUI. \end{itemize} \sphinxAtStartPar As these fields are common to all models, we omit them from the per\sphinxhyphen{}model descriptions that follow. Note that we use upper case to denote a model (e.g., \sphinxtitleref{Enterprise}) and lower case to denote a field within a model (e.g., \sphinxtitleref{enterprise}). \subsection{6.4.1 Enterprises} \label{\detokenize{cloud:enterprises}} \sphinxAtStartPar Aether is deployed in enterprises, and so needs to define a representative set of organizational abstractions. These include \sphinxtitleref{Enterprise}, which forms the root of a customer\sphinxhyphen{}specific hierarchy. The \sphinxtitleref{Enterprise} model is referenced by many other objects, and allows those objects to be scoped to a particular Enterprise for ownership and role\sphinxhyphen{}based access control purposes. \sphinxtitleref{Enterprise} contains the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{connectivity\sphinxhyphen{}service}: A list of backend subsystems that implement connectivity for this enterprise. This list corresponds to the API endpoint for the SD\sphinxhyphen{}Core, SD\sphinxhyphen{}Fabric, and SD\sphinxhyphen{}RAN components. \end{itemize} \sphinxAtStartPar \sphinxtitleref{Enterprises} are further divided into \sphinxtitleref{Sites}. A site is a point\sphinxhyphen{}of\sphinxhyphen{}presence for an \sphinxtitleref{Enterprise} and may be either physical or logical (i.e., a single geographic location could contain several logical sites). The \sphinxtitleref{Site} model, in turn, contains the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{enterprise}: A link to the \sphinxtitleref{Enterprise} that owns this site. \item {} \sphinxAtStartPar \sphinxtitleref{imsi\sphinxhyphen{}definition}: A description of how IMSIs are constructed for this site. It consists of the following sub\sphinxhyphen{}fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{mcc}: Mobile country code. \item {} \sphinxAtStartPar \sphinxtitleref{mnc}: Mobile network code. \item {} \sphinxAtStartPar \sphinxtitleref{enterprise}: A numeric enterprise id. \item {} \sphinxAtStartPar \sphinxtitleref{format}: A mask that defines how the above three fields are encoded in an IMSI. For example \sphinxtitleref{CCCNNNEEESSSSSS} specifies an IMSI with a 3\sphinxhyphen{}digit MCC, a 3\sphinxhyphen{}digit MNC, a 3\sphinxhyphen{}digit ENT, and a 6\sphinxhyphen{}digit subscriber. \end{itemize} \end{itemize} \sphinxAtStartPar As a reminder, an IMSI is burned into every SIM card, and is used to identify and locate UEs throughout the global cellular network. \subsection{6.4.2 Slices} \label{\detokenize{cloud:slices}} \sphinxAtStartPar Aether models 5G connectivity as a \sphinxtitleref{Slice}, which represents an isolated communication channel (and associated QoS parameters) that connects a set of devices (modeled as a \sphinxtitleref{Device\sphinxhyphen{}Group}) to a set of applications (each of which is modeled as an \sphinxtitleref{Application}). For example, an enterprise might configure one slice to carry IoT traffic and another slice to carry video traffic. The \sphinxtitleref{Slice} model has the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{device\sphinxhyphen{}group}: A list of \sphinxtitleref{Device\sphinxhyphen{}Group} objects that can participate in this \sphinxtitleref{Slice}. Each entry in the list contains both the reference to the \sphinxtitleref{Device\sphinxhyphen{}Group} as well as an \sphinxtitleref{enable} field which may be used to temporarily remove access to the group. \item {} \sphinxAtStartPar \sphinxtitleref{application}: A list of \sphinxtitleref{Application} objects that are either allowed or denied for this \sphinxtitleref{Slice}. Each entry in the list contains both a reference to the \sphinxtitleref{Application} as well as an \sphinxtitleref{allow} field which can be set to \sphinxtitleref{true} to allow the application or \sphinxtitleref{false} to deny it. \item {} \sphinxAtStartPar \sphinxtitleref{template}: Reference to the \sphinxtitleref{Template} that was used to initialize this \sphinxtitleref{Slice}. \item {} \sphinxAtStartPar \sphinxtitleref{upf}: Reference to the User Plane Function (\sphinxtitleref{UPF}) that should be used to process packets for this \sphinxtitleref{Slice}. Multiple \sphinxtitleref{Slices} may share a single \sphinxtitleref{UPF}. \item {} \sphinxAtStartPar \sphinxtitleref{enterprise}: Reference to the \sphinxtitleref{Enterprise} that owns this \sphinxtitleref{Slice}. \item {} \sphinxAtStartPar \sphinxtitleref{site}: Reference to the \sphinxtitleref{Site} where this \sphinxtitleref{Slice} is deployed. \item {} \sphinxAtStartPar \sphinxtitleref{sst}, \sphinxtitleref{sd}: 3GPP\sphinxhyphen{}defined slice identifiers assigned by the operations team. \item {} \sphinxAtStartPar \sphinxtitleref{mbr.uplink}, \sphinxtitleref{mbr.downlink}, \sphinxtitleref{mbr.uplink\sphinxhyphen{}burst\sphinxhyphen{}size}, \sphinxtitleref{mbr.downlink\sphinxhyphen{}burst\sphinxhyphen{}size}: Maximum bit\sphinxhyphen{}rate and burst sizes for this slice. \end{itemize} \sphinxAtStartPar The rate\sphinxhyphen{}related parameters are initialized using a selected \sphinxtitleref{template}, as described below, but these values may be changed at runtime. Also note that this example illustrates how modeling can be used to enforce invariants, in this case, that the \sphinxtitleref{Site} of the \sphinxtitleref{UPF} and \sphinxtitleref{Device\sphinxhyphen{}Group} must match the \sphinxtitleref{Site} of the \sphinxtitleref{Slice}. That is, the physical devices that connect to a slice and the UPF that implements the core segment of the slice must be constrained to a single physical location. \sphinxAtStartPar At one end of a Slice is a \sphinxtitleref{Device\sphinxhyphen{}Group}, which identifies a set of devices that are allowed to use the Slice to connect to various applications. The \sphinxtitleref{Device\sphinxhyphen{}Group} model contains the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{imsis}: A list of IMSI ranges. Each range has the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{name}: Name of the range. Used as a key. \item {} \sphinxAtStartPar \sphinxtitleref{imsi\sphinxhyphen{}range\sphinxhyphen{}from}: First IMSI in the range. \item {} \sphinxAtStartPar \sphinxtitleref{imsi\sphinxhyphen{}range\sphinxhyphen{}to}: Last IMSI in the range. Can be omitted if the range only contains one IMSI. \end{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{ip\sphinxhyphen{}domain}: Reference to an \sphinxtitleref{IP\sphinxhyphen{}Domain} object that describes the IP and DNS settings for UEs within this group. \item {} \sphinxAtStartPar \sphinxtitleref{site}: Reference to the site where this \sphinxtitleref{Device\sphinxhyphen{}Group} may be used. (This field indirectly identifies the \sphinxtitleref{Enterprise} since a \sphinxtitleref{Site} contains a reference to \sphinxtitleref{Enterprise}.) \item {} \sphinxAtStartPar \sphinxtitleref{mbr.uplink}, \sphinxtitleref{mbr.downlink}: Maximum bit\sphinxhyphen{}rate for the device group. \item {} \sphinxAtStartPar \sphinxtitleref{traffic\sphinxhyphen{}class}: The traffic class to be used for devices in this group. \end{itemize} \sphinxAtStartPar At the other end of a Slice is a list of \sphinxtitleref{Application} objects, which specifies the endpoints for the program devices talk to. The \sphinxtitleref{Application} model contains the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{address}: The DNS name or IP address of the endpoint. \item {} \sphinxAtStartPar \sphinxtitleref{endpoint}: A list of endpoints. Each has the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{name}: Name of the endpoint. Used as a key. \item {} \sphinxAtStartPar \sphinxtitleref{port\sphinxhyphen{}start}: Starting port number. \item {} \sphinxAtStartPar \sphinxtitleref{port\sphinxhyphen{}end}: Ending port number. \item {} \sphinxAtStartPar \sphinxtitleref{protocol}: Protocol (\sphinxtitleref{TCP|UDP}) for the endpoint. \item {} \sphinxAtStartPar \sphinxtitleref{mbr.uplink}, \sphinxtitleref{mbr.downlink}: Maximum bitrate for devices communicating with this application. \item {} \sphinxAtStartPar \sphinxtitleref{traffic\sphinxhyphen{}class}: Traffic class for devices communicating with this application. \end{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{enterprise}: Link to an \sphinxtitleref{Enterprise} object that owns this application. May be left empty to indicate a global application that may be used by multiple enterprises. \end{itemize} \sphinxAtStartPar Note that Aether’s \sphinxstyleemphasis{Slice} abstraction is similar to 3GPP’s specification of a “slice”, but the \sphinxtitleref{Slice} model includes a combination of 3GPP\sphinxhyphen{}specified identifiers (e.g., \sphinxtitleref{sst} and \sphinxtitleref{sd}), and details about the underlying implementation (e.g., \sphinxtitleref{upf} denotes the UPF implementation for the Core’s user plane). The \sphinxtitleref{Slice} model also includes fields related to RAN slicing, with the Service Orchestrator responsible for stitching together end\sphinxhyphen{}to\sphinxhyphen{}end connectivity across the RAN, Core, and Fabric. \subsection{6.4.3 QoS Profiles} \label{\detokenize{cloud:qos-profiles}} \sphinxAtStartPar Associated with each Slice is a QoS\sphinxhyphen{}related profile that governs how traffic carried by that slice is to be treated. This starts with a \sphinxtitleref{Template} model, which defines the valid (accepted) connectivity settings. The Aether Operations team is responsible for defining these (the features they offer must be supported by the backend subsystems), with enterprises selecting the template they want applied to any instances of the connectivity service they create (e.g., via a drop\sphinxhyphen{}down menu). That is, templates are used to initialize \sphinxtitleref{Slice} objects. The \sphinxtitleref{Template} model has the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{sst}, \sphinxtitleref{sd}: Slice identifiers, as specified by 3GPP. \item {} \sphinxAtStartPar \sphinxtitleref{mbr.uplink}, \sphinxtitleref{mbr.downlink}: Maximum uplink and downlink bandwidth. \item {} \sphinxAtStartPar \sphinxtitleref{mbr.uplink\sphinxhyphen{}burst\sphinxhyphen{}size}, \sphinxtitleref{mbr.downlink\sphinxhyphen{}burst\sphinxhyphen{}size}: Maximum burst size. \item {} \sphinxAtStartPar \sphinxtitleref{traffic\sphinxhyphen{}class}: Link to a \sphinxtitleref{Traffic\sphinxhyphen{}Class} object that describes the type of traffic. \end{itemize} \sphinxAtStartPar You will see that the \sphinxtitleref{Device\sphinxhyphen{}Group} and \sphinxtitleref{Application} models include similar fields. The idea is that QoS parameters are established for the slice as a whole (based on the selected \sphinxtitleref{Template}) and then individual devices and applications connected to that slice can define their own, more\sphinxhyphen{}restrictive QoS parameters on an instance\sphinxhyphen{}by\sphinxhyphen{}instance basis. \sphinxAtStartPar Finally, the \sphinxtitleref{Traffic\sphinxhyphen{}Class} model specifies the classes of traffic, and includes the following fields: \begin{itemize} \item {} \sphinxAtStartPar \sphinxtitleref{arp}: Allocation and retention priority. \item {} \sphinxAtStartPar \sphinxtitleref{qci}: QoS class identifier. \item {} \sphinxAtStartPar \sphinxtitleref{pelr}: Packet error loss rate. \item {} \sphinxAtStartPar \sphinxtitleref{pdb}: Packet delay budget. \end{itemize} \subsection{6.4.4 Other Models} \label{\detokenize{cloud:other-models}} \sphinxAtStartPar The above description references other models, which we do not fully describe here. They include \sphinxtitleref{AP\sphinxhyphen{}List}, which specifies a list of access points (radios); \sphinxtitleref{IP\sphinxhyphen{}Domain}, which specifies IP and DNS settings; and \sphinxtitleref{UPF}, which specifies the User Plane Function (the data plane element of the SD\sphinxhyphen{}Core) that is to forward packets on behalf of this particular instance of the connectivity service. The \sphinxtitleref{UPF} model is necessary because Aether supports two different implementations: one runs as a microservice on a server and the other runs as a P4 program loaded into the switching fabric. Both implementations are described in Chapter 5. \chapter{About The Book} \label{\detokenize{README:about-the-book}}\label{\detokenize{README::doc}} \sphinxAtStartPar Source for \sphinxstyleemphasis{Private 5G: A Systems Approach} is available on GitHub under terms of the \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{Creative Commons (CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0)} license. The community is invited to contribute corrections, improvements, updates, and new material under the same terms. While this license does not automatically grant the right to make derivative works, we are keen to discuss derivative works (such as translations) with interested parties. Please reach out to \sphinxhref{mailto:discuss@systemsapproach.org}{discuss@systemsapproach.org}. \sphinxAtStartPar If you make use of this work, the attribution should include the following information: \begin{DUlineblock}{0em} \item[] \sphinxstyleemphasis{Title: Private 5G: A Systems Approach} \item[] \sphinxstyleemphasis{Authors: Larry Peterson, Oguz Sunay, and Bruce Davie} \item[] \sphinxstyleemphasis{Source:} \sphinxurl{https://github.com/SystemsApproach/private5g} \item[] \sphinxstyleemphasis{License:} \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0} \end{DUlineblock} \sphinxAtStartPar This book incorporates introductory and background content from: \begin{DUlineblock}{0em} \item[] \sphinxstyleemphasis{Title: 5G Mobile Networks: A Systems Approach} \item[] \sphinxstyleemphasis{Authors: Larry Peterson and Oguz Sunay} \item[] \sphinxstyleemphasis{Source:} \sphinxurl{https://github.com/SystemsApproach/5g} \item[] \sphinxstyleemphasis{License:} \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0} \end{DUlineblock} \sphinxAtStartPar a version of which was also published by Morgan \& Claypool in 2020 as part of their Synthesis Lectures on Network Systems. \sphinxAtStartPar Designations used by companies to distinguish their products are often claimed as trademarks or registered trademarks. In all instances in which Systems Approach, LLC, is aware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact the appropriate companies for more complete information regarding trademarks and registration. \section{Read the Book} \label{\detokenize{README:read-the-book}} \sphinxAtStartPar This book is part of the \sphinxhref{https://www.systemsapproach.org}{Systems Approach Series}, with an online version published at \sphinxhref{https://5g.systemsapproach.org}{https://5G.systemsapproach.org}. \sphinxAtStartPar For those users looking for our earlier 5G book, \sphinxstyleemphasis{5G Mobile Networks: A Systems Approach}, published as this URL, you can still find source for it archived on \sphinxhref{https://github.com/SystemsApproach/5g}{GitHub}. This new book incorporates the background material covered in the old one, plus goes into significant detail about how Private 5G is implemented and deployed as a managed cloud service. \sphinxAtStartPar To track progress and receive notices about new versions, you can follow the project on \sphinxhref{https://www.facebook.com/Computer-Networks-A-Systems-Approach-110933578952503/}{Facebook} and \sphinxhref{https://discuss.systems/@SystemsAppr}{Mastodon}. To read a running commentary on how the Internet is evolving, follow the \sphinxhref{https://systemsapproach.substack.com}{Systems Approach on Substack}. \section{Build the Book} \label{\detokenize{README:build-the-book}} \sphinxAtStartPar To build a web\sphinxhyphen{}viewable version, you first need to download the source: \begin{sphinxVerbatim}[commandchars=\\\{\}] \PYGZdl{} mkdir \PYGZti{}/systemsapproach \PYGZdl{} cd \PYGZti{}/systemsapproach \PYGZdl{} git clone https://github.com/SystemsApproach/private5g.git \PYGZdl{} cd private5g \end{sphinxVerbatim} \sphinxAtStartPar The build process is stored in the Makefile and requires Python be installed. The Makefile will create a virtualenv (\sphinxcode{\sphinxupquote{venv\sphinxhyphen{}docs}}) which installs the documentation generation toolset. You may also need to install the \sphinxcode{\sphinxupquote{enchant}} C library using your system’s package manager for the spelling checker to function properly. \sphinxAtStartPar To generate HTML in \sphinxcode{\sphinxupquote{\_build/html}}, run \sphinxcode{\sphinxupquote{make html}}. \sphinxAtStartPar To check the formatting of the book, run \sphinxcode{\sphinxupquote{make lint}}. \sphinxAtStartPar To check spelling, run \sphinxcode{\sphinxupquote{make spelling}}. If there are additional words, names, or acronyms that are correctly spelled but not in the dictionary, please add them to the \sphinxcode{\sphinxupquote{dict.txt}} file. \sphinxAtStartPar To see the other available output formats, run \sphinxcode{\sphinxupquote{make}}. \section{Contribute to the Book} \label{\detokenize{README:contribute-to-the-book}} \sphinxAtStartPar We hope that if you use this material, you are also willing to contribute back to it. If you are new to open source, you might check out this \sphinxhref{https://opensource.guide/how-to-contribute/}{How to Contribute to Open Source} guide. Among other things, you’ll learn about posting \sphinxstyleemphasis{Issues} that you’d like to see addressed, and issuing \sphinxstyleemphasis{Pull Requests} to merge your improvements back into GitHub. \sphinxAtStartPar If you’d like to contribute and are looking for something that needs attention, see the \sphinxhref{https://github.com/SystemsApproach/private5g/wiki}{wiki} for the current TODO list. \chapter{About The Authors} \label{\detokenize{authors:about-the-authors}}\label{\detokenize{authors::doc}} \sphinxAtStartPar \sphinxstylestrong{Larry Peterson} is the Robert E. Kahn Professor of Computer Science, Emeritus at Princeton University, where he served as Chair from 2003\sphinxhyphen{}2009. His research focuses on the design, implementation, and operation of Internet\sphinxhyphen{}scale distributed systems, including the widely used PlanetLab and MeasurementLab platforms. He is currently contributing to the Aether access\sphinxhyphen{}edge cloud project at the Open Networking Foundation (ONF), where he serves as Chief Scientist. Peterson is a member of the National Academy of Engineering, a Fellow of the ACM and the IEEE, the 2010 recipient of the IEEE Kobayashi Computer and Communication Award, and the 2013 recipient of the ACM SIGCOMM Award. He received his Ph.D. degree from Purdue University. \sphinxAtStartPar \sphinxstylestrong{Oguz Sunay} is Chief Architect for the Network and Edge Group (NEX) at Intel, which he joined as part of Intel’s acquisition of the Open Networking Foundation (ONF) engineering team. While at ONF, he was Vice President for Research \& Development, where led all mobile\sphinxhyphen{}related projects. Before joining ONF, Sunay was the CTO at Argela\sphinxhyphen{}USA, where he was the innovator of a Programmable Radio Access Network Architecture (ProgRAN) for 5G that enabled the world’s first dynamically programmable RAN slicing solution. He has also held prior industry positions at Nokia Research Center and Bell Laboratories, where he focused on 3G and 4G end\sphinxhyphen{}to\sphinxhyphen{}end systems architectures and participated and chaired various standardization activities. Sunay has also spent over 10 years in academia, as a Professor of Electrical and Computer Engineering. He holds many U.S. and European patents on various aspects of 3G, 4G, and 5G, and has authored numerous journal and conference publications. He received his Ph.D. and M.Sc. from Queen’s University, Canada, and his B.Sc.Hon. from METU, Turkey. \sphinxAtStartPar \sphinxstylestrong{Bruce Davie} is a computer scientist noted for his contributions to the field of networking. He is a former VP and CTO for the Asia Pacific region at VMware. He joined VMware during the acquisition of Software Defined Networking (SDN) startup Nicira. Prior to that, he was a Fellow at Cisco Systems, leading a team of architects responsible for Multiprotocol Label Switching (MPLS). Davie has over 30 years of networking industry experience and has co\sphinxhyphen{}authored 17 RFCs. He was recognized as an ACM Fellow in 2009 and chaired ACM SIGCOMM from 2009 to 2013. He was also a visiting lecturer at the Massachusetts Institute of Technology for five years. Davie is the author of multiple books and the holder of more than 40 U.S. Patents. \renewcommand{\indexname}{Index} \printindex \end{document}