%% Generated by Sphinx. \def\sphinxdocclass{report} \documentclass[a4paper,11pt,english]{sphinxmanual} \ifdefined\pdfpxdimen \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen \fi \sphinxpxdimen=.75bp\relax \ifdefined\pdfimageresolution \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax \fi %% let collapsible pdf bookmarks panel have high depth per default \PassOptionsToPackage{bookmarksdepth=5}{hyperref} \PassOptionsToPackage{warn}{textcomp} \usepackage[utf8]{inputenc} \ifdefined\DeclareUnicodeCharacter % support both utf8 and utf8x syntaxes \ifdefined\DeclareUnicodeCharacterAsOptional \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}} \else \let\sphinxDUC\DeclareUnicodeCharacter \fi \sphinxDUC{00A0}{\nobreakspace} \sphinxDUC{2500}{\sphinxunichar{2500}} \sphinxDUC{2502}{\sphinxunichar{2502}} \sphinxDUC{2514}{\sphinxunichar{2514}} \sphinxDUC{251C}{\sphinxunichar{251C}} \sphinxDUC{2572}{\textbackslash} \fi \usepackage{cmap} \usepackage[LGR,T1]{fontenc} \usepackage{amsmath,amssymb,amstext} \usepackage{babel} \usepackage{substitutefont} \usepackage{tgtermes} \usepackage{tgheros} \renewcommand{\ttdefault}{txtt} \expandafter\ifx\csname T@LGR\endcsname\relax \else % LGR was declared as font encoding \substitutefont{LGR}{\rmdefault}{cmr} \substitutefont{LGR}{\sfdefault}{cmss} \substitutefont{LGR}{\ttdefault}{cmtt} \fi \expandafter\ifx\csname T@X2\endcsname\relax \expandafter\ifx\csname T@T2A\endcsname\relax \else % T2A was declared as font encoding \substitutefont{T2A}{\rmdefault}{cmr} \substitutefont{T2A}{\sfdefault}{cmss} \substitutefont{T2A}{\ttdefault}{cmtt} \fi \else % X2 was declared as font encoding \substitutefont{X2}{\rmdefault}{cmr} \substitutefont{X2}{\sfdefault}{cmss} \substitutefont{X2}{\ttdefault}{cmtt} \fi \usepackage{textalpha} \usepackage[Bjarne]{fncychap} \usepackage[,numfigreset=1,mathnumfig]{sphinx} \fvset{fontsize=auto} \usepackage{geometry} % Include hyperref last. \usepackage{hyperref} % Fix anchor placement for figures with captions. \usepackage{hypcap}% it must be loaded after hyperref. % Set up styles of URL: it should be placed after hyperref. \urlstyle{same} \addto\captionsenglish{\renewcommand{\contentsname}{Table of Contents}} \usepackage{sphinxmessages} \setcounter{tocdepth}{1} \title{5G Mobile Networks: A Systems Approach} \date{Dec 09, 2022} \release{Version 1.1\sphinxhyphen{}dev} \author{Peterson and Sunay} \newcommand{\sphinxlogo}{\vbox{}} \renewcommand{\releasename}{Release} \makeindex \begin{document} \pagestyle{empty} \sphinxmaketitle \pagestyle{plain} \sphinxtableofcontents \pagestyle{normal} \phantomsection\label{\detokenize{index::doc}} \chapter*{Preface} \label{\detokenize{preface:preface}}\label{\detokenize{preface::doc}} \sphinxAtStartPar The transition to 5G is happening, and unless you’ve been actively trying to ignore it, you’ve undoubtedly heard the hype. But if you are like 99\% of the CS\sphinxhyphen{}trained, systems\sphinxhyphen{}oriented, cloud\sphinxhyphen{}savvy people in the world, the cellular network is largely a mystery. You know it’s an important technology used in the last mile to connect people to the Internet, but you’ve otherwise abstracted it out of your scope\sphinxhyphen{}of\sphinxhyphen{}concerns. Perhaps you’ve heard a conspiracy theory or two about how 5G is going to cause some awful side\sphinxhyphen{}effect—a sign that the hype around 5G might not be working as intended. \sphinxAtStartPar The important thing to understand about 5G is that it implies much more than a generational upgrade in bandwidth (although it does involve that too). It involves transformative changes to the access network that will make it much more like a modern cloud. Important technology trends such as software\sphinxhyphen{}defined networking (SDN) and open source software will lead to an access network that is much more nimble and innovative. And it will enable new classes of application, particularly a broad set of IoT (Internet of Things) applications. In fact we could even see the “Access\sphinxhyphen{}as\sphinxhyphen{}frontend\sphinxhyphen{}to\sphinxhyphen{}Internet” perspective being turned on its head, as the access network itself becomes a collection of clouds with new services delivered directly from “edge clouds”. \sphinxAtStartPar This book is written for someone who has a working understanding of the Internet and the cloud, but has had limited success penetrating the myriad of acronyms that dominate cellular networking. In fairness, the Internet has its share of acronyms, but it also comes with a sufficient set of abstractions to help manage the complexity. It’s hard to say the same for the cellular network, where pulling on one thread seemingly unravels the entire space. It has also been the case that the cellular network had been largely hidden inside proprietary devices, which has made it impossible to figure it out for yourself. \sphinxAtStartPar This book is the result of a mobile networking expert teaching a systems person about 5G as we’ve collaborated on an open source 5G implementation. The material has been used to train other software developers, and we are hopeful it will be useful to anyone who wants a deeper understanding of 5G and the opportunity for innovation it provides. Readers who want hands\sphinxhyphen{}on experience can also access the open source software introduced in the book. \sphinxAtStartPar This book will likely be a work\sphinxhyphen{}in\sphinxhyphen{}progress for the foreseeable future. It’s not intended to be encyclopedic—favoring perspective and end\sphinxhyphen{}to\sphinxhyphen{}end completeness over every last bit of detail—but we do plan to flesh out the content over time. Your suggestions (and contributions) to this end are welcome. \section*{Acknowledgements} \label{\detokenize{preface:acknowledgements}} \sphinxAtStartPar The software described in this book is due to the hard work of the ONF engineering team and the open source community that works with them. We acknowledge their contributions. We also thank Bruce Davie, Guru Parulkar, and Changhoon Kim for their feedback on early drafts of the manuscript. \begin{DUlineblock}{0em} \item[] Larry Peterson and Oguz Sunay \item[] Open Networking Foundation \item[] March 2020 \end{DUlineblock} \chapter{Chapter 1: Introduction} \label{\detokenize{intro:chapter-1-introduction}}\label{\detokenize{intro::doc}} \sphinxAtStartPar Mobile networks, which have a 40\sphinxhyphen{}year history that parallels the Internet’s, have undergone significant change. The first two generations supported voice and then text, with 3G defining the transition to broadband access, supporting data rates measured in hundreds of kilobits\sphinxhyphen{}per\sphinxhyphen{}second. Today, the industry is at 4G (supporting data rates typically measured in the few megabits\sphinxhyphen{}per\sphinxhyphen{}second) and transitioning to 5G, with the promise of a tenfold increase in data rates. \sphinxAtStartPar But 5G is about much more than increased bandwidth. 5G represents a fundamental rearchitecting of the access network in a way that leverages several key technology trends and sets it on a path to enable much greater innovation. In the same way that 3G defined the transition from voice to broadband, 5G’s promise is primarily about the transition from a single access service (broadband connectivity) to a richer collection of edge services and devices. 5G is expected to provide support for immersive user interfaces (e.g., AR/VR), mission\sphinxhyphen{}critical applications (e.g., public safety, autonomous vehicles), and the Internet\sphinxhyphen{}of\sphinxhyphen{}Things (IoT). Because these use cases will include everything from home appliances to industrial robots to self\sphinxhyphen{}driving cars, 5G won’t just support humans accessing the Internet from their smartphones, but also swarms of autonomous devices working together on their behalf. There is more to supporting these services than just improving bandwidth or latency to individual users. As we will see, a fundamentally different edge network architecture is required. \sphinxAtStartPar The requirements for this architecture are ambitious, and can be illustrated by three classes of capabilities: \begin{itemize} \item {} \sphinxAtStartPar To support \sphinxstyleemphasis{Massive Internet\sphinxhyphen{}of\sphinxhyphen{}Things}, potentially including devices with ultra\sphinxhyphen{}low energy (10+ years of battery life), ultra\sphinxhyphen{}low complexity (10s of bits\sphinxhyphen{}per\sphinxhyphen{}second), and ultra\sphinxhyphen{}high density (1 million nodes per square kilometer). \item {} \sphinxAtStartPar To support \sphinxstyleemphasis{Mission\sphinxhyphen{}Critical Control}, potentially including ultra\sphinxhyphen{}high availability (greater than 99.999\% or “five nines”), ultra\sphinxhyphen{}low latency (as low as 1 ms), and extreme mobility (up to 100 km/h). \item {} \sphinxAtStartPar To support \sphinxstyleemphasis{Enhanced Mobile Broadband}, potentially including extreme data rates (multi\sphinxhyphen{}Gbps peak, 100+ Mbps sustained) and extreme capacity (10 Tbps of aggregate throughput per square kilometer). \end{itemize} \sphinxAtStartPar These targets will certainly not be met overnight, but that’s in keeping with each generation of the mobile network being a decade\sphinxhyphen{}long endeavor. \sphinxAtStartPar On top of these quantitative improvements to the capabilities of the access network, 5G is being viewed as a chance for building a platform to support innovation. Whereas prior access networks were generally optimized for known services (such as voice calls and SMS), the Internet has been hugely successful in large part because it supported a wide range of applications that were not even thought of when it was first designed. The 5G network is very much being designed with this same goal of enabling all sorts of future applications beyond those we fully recognize today. \phantomsection\label{\detokenize{intro:reading-vision}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For an example of the grand vision for 5G from one of the industry leaders, see \sphinxhref{https://www.qualcomm.com/media/documents/files/whitepaper-making-5g-nr-a-reality.pdf}{Making 5G NR a Reality}. Qualcomm Whitepaper, December 2016. \end{sphinxadmonition} \sphinxAtStartPar The 5G mobile network, because it is on an evolutionary path and not a point solution, includes standardized specifications, a range of implementation choices, and a long list of aspirational goals. Because this leaves so much room for interpretation, our approach to describing 5G is grounded in two mutually supportive principles. The first is to apply a \sphinxstyleemphasis{systems lens}, which is to say, we explain the sequence of design decisions that lead to a solution rather than fall back on enumerating the overwhelming number of acronyms or individual point technologies as a \sphinxstyleemphasis{fait accompli}. The second is to aggressively disaggregate the system. Building a disaggregated, virtualized, and software\sphinxhyphen{}defined 5G access network is the direction the industry is already headed (for good technical and business reasons), but breaking the 5G network down into its elemental components is also the best way to explain how 5G works. It also helps to illustrate how 5G might evolve in the future to provide even more value. \begin{sphinxShadowBox} \sphinxstylesidebartitle{Evolutionary Path} \sphinxAtStartPar That 5G is on an evolutionary path is the central theme of this book. We call attention to its importance here, and revisit the topic throughout the book. \sphinxAtStartPar We are writing this book for \sphinxstyleemphasis{system generalists}, with the goal of helping bring a community that understands a broad range of systems issues (but knows little or nothing about the cellular network) up to speed so they can play a role in its evolution. This is a community that understands both feature velocity and best practices in building robust scalable systems, and so has an important role to play in bringing all of 5G’s potential to fruition. \end{sphinxShadowBox} \sphinxAtStartPar What this all means is that there is no single, comprehensive definition of 5G, any more than there is for the Internet. It is a complex and evolving system, constrained by a set of standards that purposely give all the stakeholders many degrees of freedom. In the chapters that follow, it should be clear from the context whether we are talking about \sphinxstyleemphasis{standards} (what everyone must do to interoperate), \sphinxstyleemphasis{trends} (where the industry seems to be headed), or \sphinxstyleemphasis{implementation choices} (examples to make the discussion more concrete). By adopting a systems perspective throughout, our intent is to describe 5G in a way that helps the reader navigate this rich and rapidly evolving system. \section{1.1 Standardization Landscape} \label{\detokenize{intro:standardization-landscape}} \sphinxAtStartPar As of 3G, the generational designation corresponds to a standard defined by the \sphinxstyleemphasis{3rd Generation Partnership Project (3GPP)}. Even though its name has “3G” in it, the 3GPP continues to define the standards for 4G and 5G, each of which corresponds to a sequence of releases of the standard. Release 15 is considered the demarcation point between 4G and 5G, with Release 17 scheduled for 2021. Complicating the terminology, 4G was on a multi\sphinxhyphen{}release evolutionary path referred to as \sphinxstyleemphasis{Long Term Evolution (LTE)}. 5G is on a similar evolutionary path, with several expected releases over its lifetime. \sphinxAtStartPar While 5G is an ambitious advance beyond 4G, it is also the case that understanding 4G is the first step to understanding 5G, as several aspects of the latter can be explained as bringing a new degree\sphinxhyphen{}of\sphinxhyphen{}freedom to the former. In the chapters that follow, we often introduce some architectural feature of 4G as a way of laying the foundation for the corresponding 5G component. \sphinxAtStartPar Like Wi\sphinxhyphen{}Fi, cellular networks transmit data at certain bandwidths in the radio spectrum. Unlike Wi\sphinxhyphen{}Fi, which permits anyone to use a channel at either 2.4 or 5 GHz (these are unlicensed bands), governments have auctioned off and licensed exclusive use of various frequency bands to service providers, who in turn sell mobile access service to their subscribers. \sphinxAtStartPar There is also a shared\sphinxhyphen{}license band at 3.5 GHz, called \sphinxstyleemphasis{Citizens Broadband Radio Service (CBRS)}, set aside in North America for cellular use. Similar spectrum is being set aside in other countries. The CBRS band allows 3 tiers of users to share the spectrum: first right of use goes to the original owners of this spectrum (naval radars and satellite ground stations); followed by priority users who receive this right over 10MHz bands for three years via regional auctions; and finally the rest of the population, who can access and utilize a portion of this band as long as they first check with a central database of registered users. CBRS, along with standardization efforts to extend cellular networks to operate in the unlicensed bands, open the door for private cellular networks similar to Wi\sphinxhyphen{}Fi. \sphinxAtStartPar The specific frequency bands that are licensed for cellular networks vary around the world, and are complicated by the fact that network operators often simultaneously support both old/legacy technologies and new/next\sphinxhyphen{}generation technologies, each of which occupies a different frequency band. The high\sphinxhyphen{}level summary is that traditional cellular technologies range from 700\sphinxhyphen{}2400 MHz, with new mid\sphinxhyphen{}spectrum allocations now happening at 6 GHz, and millimeter\sphinxhyphen{}wave (mmWave) allocations opening above 24 GHz. \sphinxAtStartPar While the specific frequency band is not directly relevant to understanding 5G from an architectural perspective, it does impact the physical\sphinxhyphen{}layer components, which in turn has indirect ramifications on the overall 5G system. We identify and explain these ramifications in later chapters. Ensuring that the allocated spectrum is used \sphinxstyleemphasis{efficiently} is also a critical design goal. \section{1.2 Access Networks} \label{\detokenize{intro:access-networks}} \sphinxAtStartPar The cellular network is part of the access network that implements the Internet’s so\sphinxhyphen{}called \sphinxstyleemphasis{last mile}. Other access technologies include \sphinxstyleemphasis{Passive Optical Networks (PON)}, colloquially known as Fiber\sphinxhyphen{}to\sphinxhyphen{}the\sphinxhyphen{}Home. These access networks are provided by both big and small network operators. Global network operators like AT\&T run access networks at thousands of aggregation points\sphinxhyphen{}of\sphinxhyphen{}presence across a country like the US, along with a national backbone that interconnects those sites. Small regional and municipal network operators might run an access network with one or two points\sphinxhyphen{}of\sphinxhyphen{}presence, and then connect to the rest of the Internet through some large operator’s backbone. \sphinxAtStartPar In either case, access networks are physically anchored at thousands of aggregation points\sphinxhyphen{}of\sphinxhyphen{}presence within close proximity to end users, each of which serves anywhere from 1,000\sphinxhyphen{}100,000 subscribers, depending on population density. In practice, the physical deployment of these “edge” locations vary from operator to operator, but one possible scenario is to anchor both the cellular and wireline access networks in Telco \sphinxstyleemphasis{Central Offices}. \sphinxAtStartPar Historically, the Central Office—officially known as the \sphinxstyleemphasis{PSTN (Public Switched Telephone Network) Central Office}—anchored wired access (both telephony and broadband), while the cellular network evolved independently by deploying a parallel set of \sphinxstyleemphasis{Mobile Telephone Switching Offices (MTSO)}. Each MTSO serves as a \sphinxstyleemphasis{mobile aggregation} point for the set of cell towers in a given geographic area. For our purposes, the important idea is that such aggregation points exist, and it is reasonable to think of them as defining the edge of the operator\sphinxhyphen{}managed access network. For simplicity, we sometimes use the term “Central Office” as a synonym for both types of edge sites. \section{1.3 Edge Cloud} \label{\detokenize{intro:edge-cloud}} \sphinxAtStartPar Because of their wide distribution and close proximity to end users, Central Offices are also an ideal place to host the edge cloud. But this raises the question: What exactly is the edge cloud? \sphinxAtStartPar In a nutshell, the cloud began as a collection of warehouse\sphinxhyphen{}sized datacenters, each of which provided a cost\sphinxhyphen{}effective way to power, cool, and operate a scalable number of servers. Over time, this shared infrastructure lowered the barrier to deploying scalable Internet services, but today, there is increasing pressure to offer low\sphinxhyphen{}latency/high\sphinxhyphen{}bandwidth cloud applications that cannot be effectively implemented in centralized datacenters. Augmented Reality (AR), Virtual Reality (VR), Internet\sphinxhyphen{}of\sphinxhyphen{}Things (IoT), and Autonomous Vehicles are all examples of this kind of application. This has resulted in a trend to move some functionality out of the datacenter and towards the edge of the network, closer to end users. \sphinxAtStartPar Where this edge is \sphinxstyleemphasis{physically} located depends on who you ask. If you ask a network operator that already owns and operates thousands of Central Offices, then their Central Offices are an obvious answer. Others might claim the edge is located at the 14,000 Starbucks across the US, and still others might point to the tens\sphinxhyphen{}of\sphinxhyphen{}thousands of cell towers spread across the globe. \sphinxAtStartPar Our approach is to be location agnostic, but it is worth pointing out that the cloud’s migration to the edge coincides with a second trend, which is that network operators are re\sphinxhyphen{}architecting the access network to use the same commodity hardware and best practices in building scalable software as the cloud providers. Such a design, which is sometimes referred to as CORD \sphinxstyleemphasis{(Central Office Re\sphinxhyphen{}architected as a Datacenter)}, supports both the access network and edge services co\sphinxhyphen{}located on a shared cloud platform. This platform is then replicated across hundreds or thousands of sites (including, but not limited to, Central Offices). So while we shouldn’t limit ourselves to the Central Office as the only answer to the question of where the edge cloud is located, it is becoming a viable option. \phantomsection\label{\detokenize{intro:reading-cord}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar To learn about the technical origins of CORD, which was first applied to fiber\sphinxhyphen{}based access networks (PON), see \sphinxhref{https://wiki.opencord.org/download/attachments/1278027/PETERSON\_CORD.pdf}{Central Office Re\sphinxhyphen{}architected as a Datacenter, IEEE Communications, October 2016}. \sphinxAtStartPar To understand the business case for CORD (and CORD\sphinxhyphen{}inspired technologies), see the A.D. Little report \sphinxhref{https://www.adlittle.com/en/who-dares-wins}{Who Dares Wins! How Access Transformation Can Fast\sphinxhyphen{}Track Evolution of Operator Production Platforms, September 2019}. \end{sphinxadmonition} \sphinxAtStartPar When we get into the details of how 5G can be implemented in practice, we use CORD as our exemplar. For now, the important thing to understand is that 5G is being implemented as software running on commodity hardware, rather than embedded in the special\sphinxhyphen{}purpose proprietary hardware used in past generations. This has a significant impact on how we think about 5G (and how we describe 5G), which will increasingly become yet another software\sphinxhyphen{}based component in the cloud, as opposed to an isolated and specialized technology attached to the periphery of the cloud. \sphinxAtStartPar Keep in mind that our use of CORD as an exemplar is not to imply that the edge cloud is limited to Central Offices. CORD is a good exemplar because it is designed to host both edge services and access technologies like 5G on a common platform, where the Telco Central Office is one possible location to deploy such a platform. \sphinxAtStartPar An important takeaway from this discussion is that to understand how 5G is being implemented, it is helpful to have a working understanding of how clouds are built. This includes the use of \sphinxstyleemphasis{commodity hardware} (both servers and white\sphinxhyphen{}box switches), horizontally scalable \sphinxstyleemphasis{microservices} (also referred to as \sphinxstyleemphasis{cloud native}), and \sphinxstyleemphasis{Software\sphinxhyphen{}Defined Networks (SDN)}. It is also helpful to have an appreciation for how cloud software is developed, tested, deployed, and operated, including practices like \sphinxstyleemphasis{DevOps} and \sphinxstyleemphasis{Continuous Integration / Continuous Deployment (CI/CD)}. \phantomsection\label{\detokenize{intro:reading-devops}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar If you are unfamiliar with SDN, we recommend a companion book: \sphinxhref{https://sdn.systemsapproach.org/}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. March 2020. \sphinxAtStartPar If you are unfamiliar with DevOps—or more generally, with the operational issues cloud providers face—we recommend \sphinxhref{https://landing.google.com/sre/books/}{Site Reliability Engineering: How Google Runs Production Systems}. \end{sphinxadmonition} \sphinxAtStartPar One final note about terminology. Anyone who has been paying attention to the discussion surrounding 5G will have undoubtedly heard about \sphinxstyleemphasis{Network Function Virtualization (NFV)}, which involves moving functionality that was once embedded in hardware appliances into VMs (or, more recently, containers) running on commodity servers. In our experience, NFV is a stepping stone towards the fully disaggregated and cloud native solution we describe in this book, and so we do not dwell on it. You can think of the NFV initiative as mostly consistent with the approach taken in this book, but making some specific engineering choices that may differ in detail from that described here. \sphinxAtStartPar While equating NFV with an implementation choice is perfectly valid, there is another interpretation of events that better captures the essence of the transformation currently underway. When Telcos began the NFV initiative, they imagined incorporating cloud technologies into their networks, creating a so\sphinxhyphen{}called \sphinxstyleemphasis{Telco Cloud}. What is actually happening instead, is that the Telco’s access technology is being subsumed into the cloud, running as yet another cloud\sphinxhyphen{}hosted workload. It would be more accurate to refer to the resulting system now emerging as the \sphinxstyleemphasis{Cloud\sphinxhyphen{}based Telco}. One reading of this book is as a roadmap to such an outcome. \chapter{Chapter 2: Radio Transmission} \label{\detokenize{primer:chapter-2-radio-transmission}}\label{\detokenize{primer::doc}} \sphinxAtStartPar For anyone familiar with wireless access technologies like Wi\sphinxhyphen{}Fi, the cellular network is most unique due to its approach to sharing the available radio spectrum among its many users, all the while allowing those users to remain connected while moving. This has resulted in a highly dynamic and adaptive approach, in which coding, modulation and scheduling play a central role. \sphinxAtStartPar As we will see in this chapter, cellular networks use a reservation\sphinxhyphen{}based strategy, whereas Wi\sphinxhyphen{}Fi is contention\sphinxhyphen{}based. This difference is rooted in each system’s fundamental assumption about utilization: Wi\sphinxhyphen{}Fi assumes a lightly loaded network (and hence optimistically transmits when the wireless link is idle and backs off if contention is detected), while 4G and 5G cellular networks assume (and strive for) high utilization (and hence explicitly assign different users to different “shares” of the available radio spectrum). \sphinxAtStartPar We start by giving a short primer on radio transmission as a way of laying a foundation for understanding the rest of the 5G architecture. The following is not a substitute for a theoretical treatment of the topic, but is instead intended as a way of grounding the systems\sphinxhyphen{}oriented description of 5G that follows in the reality of wireless communication. \section{2.1 Coding and Modulation} \label{\detokenize{primer:coding-and-modulation}} \sphinxAtStartPar The mobile channel over which digital data needs to be reliably transmitted brings a number of impairments, including noise, attenuation, distortion, fading, and interference. This challenge is addressed by a combination of coding and modulation, as depicted in \hyperref[\detokenize{primer:fig-modulation}]{Figure \ref{\detokenize{primer:fig-modulation}}}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide09}.png} \caption{The role of coding and modulation in mobile communication.}\label{\detokenize{primer:id5}}\label{\detokenize{primer:fig-modulation}}\end{figure} \sphinxAtStartPar At its core, coding inserts extra bits into the data to help recover from all the environmental factors that interfere with signal propagation. This typically implies some form of \sphinxstyleemphasis{Forward Error Correction} (e.g., turbo codes, polar codes). Modulation then generates signals that represent the encoded data stream, and it does so in a way that matches the channel characteristics: It first uses a digital modulation signal format that maximizes the number of reliably transmitted bits every second based on the specifics of the observed channel impairments; it next matches the transmission bandwidth to channel bandwidth using pulse shaping; and finally, it uses RF modulation to transmit the signal as an electromagnetic wave over an assigned \sphinxstyleemphasis{carrier frequency.} \sphinxAtStartPar For a deeper appreciation of the challenges of reliably transmitting data by propagating radio signals through the air, consider the scenario depicted in \hyperref[\detokenize{primer:fig-multipath}]{Figure \ref{\detokenize{primer:fig-multipath}}}, where the signal bounces off various stationary and moving objects, following multiple paths from the transmitter to the receiver, who may also be moving. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide10}.png} \caption{Signals propagate along multiple paths from transmitter to receiver.}\label{\detokenize{primer:id6}}\label{\detokenize{primer:fig-multipath}}\end{figure} \sphinxAtStartPar As a consequence of these multiple paths, the original signal arrives at the receiver spread over time, as illustrated in \hyperref[\detokenize{primer:fig-coherence}]{Figure \ref{\detokenize{primer:fig-coherence}}}. Empirical evidence shows that the Multipath Spread—the time between the first and last signals of one transmission arriving at the receiver—is 1\sphinxhyphen{}10μs in urban environments and 10\sphinxhyphen{}30μs in suburban environments. These multipath signals can interfere with each other constructively or destructively, and this will vary over time. Theoretical bounds for the time duration for which the channel may be assumed to be invariant, known as the \sphinxstyleemphasis{Coherence Time} and denoted \(T_c\), is given by \begin{equation*} \begin{split}T_c =c/v \times 1/f\end{split} \end{equation*} \sphinxAtStartPar where \(c\) is the velocity of the signal, \(v\) is the velocity of the receiver (e.g., moving car or train), and \(f\) is the frequency of the carrier signal that is being modulated. This says the coherence time is inversely proportional to the frequency of the signal and the speed of movement, which makes intuitive sense: The higher the frequency (narrower the wave) the shorter the coherence time, and likewise, the faster the receiver is moving the shorter the coherence time. Based on the target parameters to this model (selected according to the target physical environment), it is possible to calculate \(T_c\), which in turn bounds the rate at which symbols can be transmitted without undue risk of interference. The dynamic nature of the wireless channel is a central challenge to address in the cellular network. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide11}.png} \caption{Received data spread over time due to multipath variation.}\label{\detokenize{primer:id7}}\label{\detokenize{primer:fig-coherence}}\end{figure} \sphinxAtStartPar To complicate matters further, \hyperref[\detokenize{primer:fig-multipath}]{Figure \ref{\detokenize{primer:fig-multipath}}} and \hyperref[\detokenize{primer:fig-coherence}]{\ref{\detokenize{primer:fig-coherence}}} imply the transmission originates from a single antenna, but cell towers are equipped with an array of antennas, each transmitting in a different (but overlapping) direction. This technology, called \sphinxstyleemphasis{Multiple\sphinxhyphen{}Input\sphinxhyphen{}Multiple\sphinxhyphen{}Output (MIMO)}, opens the door to purposely transmitting data from multiple antennas in an effort to reach the receiver, adding even more paths to the environment\sphinxhyphen{}imposed multipath propagation. \sphinxAtStartPar One of the most important consequences of these factors is that the transmitter must receive feedback from every receiver to judge how to best utilize the wireless medium on their behalf. 3GPP specifies a \sphinxstyleemphasis{Channel Quality Indicator (CQI)} for this purpose, where in practice the receiver sends a CQI status report to the base station periodically (e.g., every millisecond in LTE). These CQI messages report the observed signal\sphinxhyphen{}to\sphinxhyphen{}noise ratio, which impacts the receiver’s ability to recover the data bits. The base station then uses this information to adapt how it allocates the available radio spectrum to the subscribers it is serving, as well as which coding and modulation scheme to employ. All of these decisions are made by the scheduler. \section{2.2 Scheduler} \label{\detokenize{primer:scheduler}} \sphinxAtStartPar How the scheduler does its job is one of the most important properties of each generation of the cellular network, which in turn depends on the multiplexing mechanism. For example, 2G used \sphinxstyleemphasis{Time Division Multiple Access (TDMA)} and 3G used \sphinxstyleemphasis{Code Division Multiple Access (CDMA)}. How data is multiplexed is also a major differentiator for 4G and 5G, completing the transition from the cellular network being fundamentally circuit\sphinxhyphen{}switched to fundamentally packet\sphinxhyphen{}switched. \sphinxAtStartPar Both 4G and 5G are based on \sphinxstyleemphasis{Orthogonal Frequency\sphinxhyphen{}Division Multiplexing (OFDM)}, an approach that multiplexes data over multiple orthogonal subcarrier frequencies, each of which is modulated independently. The value and efficiency of OFDM is in how it selects subcarrier frequencies so as to avoid interference, that is, how it achieves orthogonality. That topic is beyond the scope of this book. We instead take a decidedly abstract perspective of multiplexing, focusing on “discrete schedulable units of the radio spectrum” rather than the signalling and modulation underpinnings that yield those schedulable units. \sphinxAtStartPar To start, we drill down on these schedulable units. We return to the broader issue of the \sphinxstyleemphasis{air interface} that makes efficient use of the spectrum in the concluding section. \subsection{Multiplexing in 4G} \label{\detokenize{primer:multiplexing-in-4g}} \sphinxAtStartPar The 4G approach to multiplexing downstream transmissions is called \sphinxstyleemphasis{Orthogonal Frequency\sphinxhyphen{}Division Multiple Access (OFDMA)}, a specific application of OFDM that multiplexes data over a set of 12 orthogonal subcarrier frequencies, each of which is modulated independently.% \begin{footnote}[1]\sphinxAtStartFootnote 4G uses a different multiplexing strategy for upstream transmissions (from user devices to base stations), but we do not describe it because the approach is not applicable to 5G. % \end{footnote} The “Multiple Access” in OFDMA implies that data can simultaneously be sent on behalf of multiple users, each on a different subcarrier frequency and for a different duration of time. The subbands are narrow (e.g., 15 kHz), but the coding of user data into OFDMA symbols is designed to minimize the risk of data loss due to interference between adjacent bands. \sphinxAtStartPar The use of OFDMA naturally leads to conceptualizing the radio spectrum as a 2\sphinxhyphen{}D resource, as shown in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}. The minimal schedulable unit, called a \sphinxstyleemphasis{Resource Element (RE)}, corresponds to a 15\sphinxhyphen{}kHz band around one subcarrier frequency and the time it takes to transmit one OFDMA symbol. The number of bits that can be encoded in each symbol depends on the modulation rate, so for example using \sphinxstyleemphasis{Quadrature Amplitude Modulation (QAM)}, 16\sphinxhyphen{}QAM yields 4 bits per symbol and 64\sphinxhyphen{}QAM yields 6 bits per symbol. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide12}.png} \caption{Spectrum abstractly represented by a 2\sphinxhyphen{}D grid of schedulable Resource Elements.}\label{\detokenize{primer:id8}}\label{\detokenize{primer:fig-sched-grid}}\end{figure} \sphinxAtStartPar A scheduler allocates some number of REs to each user that has data to transmit during each 1 ms \sphinxstyleemphasis{Transmission Time Interval (TTI)}, where users are depicted by different colored blocks in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}. The only constraint on the scheduler is that it must make its allocation decisions on blocks of 7x12=84 resource elements, called a \sphinxstyleemphasis{Physical Resource Block (PRB)}. \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}} shows two back\sphinxhyphen{}to\sphinxhyphen{}back PRBs. Of course time continues to flow along one axis, and depending on the size of the available frequency band (e.g., it might be 100 MHz wide), there may be many more subcarrier slots (and hence PRBs) available along the other axis, so the scheduler is essentially preparing and transmitting a sequence of PRBs. \sphinxAtStartPar Note that OFDMA is not a coding/modulation algorithm, but instead provides a framework for selecting a specific coding and modulator for each subcarrier frequency. QAM is one common example modulator. It is the scheduler’s responsibility to select the modulation to use for each PRB, based on the CQI feedback it has received. The scheduler also selects the coding on a per\sphinxhyphen{}PRB basis, for example, by how it sets the parameters to the turbo code algorithm. \sphinxAtStartPar The 1\sphinxhyphen{}ms TTI corresponds to the time frame in which the scheduler receives feedback from users about the quality of the signal they are experiencing. This is the CQI mentioned earlier, where once every millisecond, each user sends a set of metrics, which the scheduler uses to make its decision as to how to allocate PRBs during the subsequent TTI. \sphinxAtStartPar Another input to the scheduling decision is the \sphinxstyleemphasis{QoS Class Identifier (QCI)}, which indicates the quality\sphinxhyphen{}of\sphinxhyphen{}service each class of traffic is to receive. In 4G, the QCI value assigned to each class (there are nine such classes, in total) indicates whether the traffic has a \sphinxstyleemphasis{Guaranteed Bit Rate (GBR)} or not \sphinxstyleemphasis{(non\sphinxhyphen{}GBR)}, plus the class’s relative priority within those two categories. \sphinxAtStartPar Finally, keep in mind that \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}} focuses on scheduling transmissions from a single antenna, but the MIMO technology described above means the scheduler also has to determine which antenna (or more generally, what subset of antennas) will most effectively reach each receiver. But again, in the abstract, the scheduler is charged with allocating a sequence of Resource Elements. \sphinxAtStartPar This all raises the question: How does the scheduler decide which set of users to service during a given time interval, how many resource elements to allocate to each such user, how to select the coding and modulation levels, and which antenna to transmit their data on? This is an optimization problem that, fortunately, we are not trying to solve here. Our goal is to describe an architecture that allows someone else to design and plug in an effective scheduler. Keeping the cellular architecture open to innovations like this is one of our goals, and as we will see in the next section, becomes even more important in 5G where the scheduler operates with even more degrees of freedom. \subsection{Multiplexing in 5G} \label{\detokenize{primer:multiplexing-in-5g}} \sphinxAtStartPar The transition from 4G to 5G introduces additional flexibility in how the radio spectrum is scheduled, making it possible to adapt the cellular network to a more diverse set of devices and applications domains. \sphinxAtStartPar Fundamentally, 5G defines a family of waveforms—unlike LTE, which specified only one waveform—each optimized for a different band in the radio spectrum.% \begin{footnote}[2]\sphinxAtStartFootnote A waveform is the frequency, amplitude, and phase\sphinxhyphen{}shift independent property (shape) of a signal. A sine wave is an example waveform. % \end{footnote} The bands with carrier frequencies below 1 GHz are designed to deliver mobile broadband and massive IoT services with a primary focus on range. Carrier frequencies between 1\sphinxhyphen{}6 GHz are designed to offer wider bandwidths, focusing on mobile broadband and mission\sphinxhyphen{}critical applications. Carrier frequencies above 24 GHz (mmWaves) are designed to provide super wide bandwidths over short, line\sphinxhyphen{}of\sphinxhyphen{}sight coverage. \sphinxAtStartPar These different waveforms affect the scheduling and subcarrier intervals (i.e., the “size” of the resource elements described in the previous section). \begin{itemize} \item {} \sphinxAtStartPar For sub\sphinxhyphen{}1 GHz bands, 5G allows maximum 50 MHz bandwidths. In this case, there are two waveforms: one with subcarrier spacing of 15 kHz and another of 30 kHz. (We used 15 kHz in the example shown in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}.) The corresponding scheduling intervals are 0.5 and 0.25 ms, respectively. (We used 0.5 ms in the example shown in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}.) \item {} \sphinxAtStartPar For 1\sphinxhyphen{}6 GHz bands, maximum bandwidths go up to 100 MHz. Correspondingly, there are three waveforms with subcarrier spacings of 15, 30 and 60 kHz, corresponding to scheduling intervals of 0.5, 0.25, and 0.125 ms, respectively. \item {} \sphinxAtStartPar For millimeter bands, bandwidths may go up to 400 MHz. There are two waveforms, with subcarrier spacings of 60 kHz and 120 kHz. Both have scheduling intervals of 0.125 ms. \end{itemize} \sphinxAtStartPar These various configurations of subcarrier spacing and scheduling intervals are sometimes called the \sphinxstyleemphasis{numerology} of the radio’s air interface. \sphinxAtStartPar This range of numerology is important because it adds another degree of freedom to the scheduler. In addition to allocating radio resources to users, it has the ability to dynamically adjust the size of the resource by changing the wave form being used. With this additional freedom, fixed\sphinxhyphen{}sized REs are no longer the primary unit of resource allocation. We instead use more abstract terminology, and talk about allocating \sphinxstyleemphasis{Resource Blocks} to subscribers, where the 5G scheduler determines both the size and number of Resource Blocks allocated during each time interval. \sphinxAtStartPar \hyperref[\detokenize{primer:fig-scheduler}]{Figure \ref{\detokenize{primer:fig-scheduler}}} depicts the role of the scheduler from this more abstract perspective, where just as with 4G, CQI feedback from the receivers and the QCI quality\sphinxhyphen{}of\sphinxhyphen{}service class selected by the subscriber are the two key pieces of input to the scheduler. Note that the set of QCI values changes between 4G and 5G, reflecting the increasing differentiation being supported. For 5G, each class includes the following attributes: \begin{itemize} \item {} \sphinxAtStartPar Resource Type: Guaranteed Bit Rate (GBR), Delay\sphinxhyphen{}Critical GBR, Non\sphinxhyphen{}GBR \item {} \sphinxAtStartPar Priority Level \item {} \sphinxAtStartPar Packet Delay Budget \item {} \sphinxAtStartPar Packet Error Rate \item {} \sphinxAtStartPar Averaging Window \item {} \sphinxAtStartPar Maximum Data Burst \end{itemize} \sphinxAtStartPar Note that while the preceding discussion could be interpreted to imply a one\sphinxhyphen{}to\sphinxhyphen{}one relationship between subscribers and a QCI, it is more accurate to say that each QCI is associated with a class of traffic (often corresponding to some type of application), where a given subscriber might be sending and receiving traffic that belongs to multiple classes at any given time. We explore this idea in much more depth in a later chapter. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide13}.png} \caption{Scheduler allocates Resource Blocks to user data streams based on CQI feedback from receivers and the QCI parameters associated with each class of service.}\label{\detokenize{primer:id9}}\label{\detokenize{primer:fig-scheduler}}\end{figure} \section{2.3 New Radio (NR)} \label{\detokenize{primer:new-radio-nr}} \sphinxAtStartPar We conclude by noting that while the previous section describes 5G as introducing additional degrees of freedom into how data is scheduled for transmission, the end result is a qualitatively more powerful radio. This new 5G air interface specification, which is commonly referred to as \sphinxstyleemphasis{New Radio (NR)}, enables three new use cases that go well beyond simply delivering increased bandwidth: \begin{itemize} \item {} \sphinxAtStartPar Extreme Mobile Broadband \item {} \sphinxAtStartPar Ultra\sphinxhyphen{}Reliable Low\sphinxhyphen{}Latency Communications \item {} \sphinxAtStartPar Massive Machine\sphinxhyphen{}Type Communications \end{itemize} \sphinxAtStartPar All three correspond to the requirements introduced in Chapter 1, and can be attributed to four fundamental improvements in how 5G multiplexes data onto the radio spectrum. \sphinxAtStartPar The first is the one identified in the previous section: being able to change the waveform. This effectively introduces the ability to dynamically change the size and number of schedulable resource units, which opens the door to making fine\sphinxhyphen{}grain scheduling decisions that are critical to predictable, low\sphinxhyphen{}latency communication. \sphinxAtStartPar The second is related to the “Multiple Access” aspect of how distinct traffic sources are multiplexed onto the available spectrum. In 4G, multiplexing happens in both the frequency and time domains for downstream traffic (as described in Section 2.2), but multiplexing happens in only the frequency domain for upstream traffic. 5G NR multiplexes both upstream and downstream traffic in both the time and frequency domains. Doing so provides finer\sphinxhyphen{}grain scheduling control needed by latency\sphinxhyphen{}sensitive applications. \sphinxAtStartPar The third is related to the new spectrum available to 5G NR, with the mmWave allocations opening above 24 GHz being especially important. This is not only because of the abundance of capacity—which makes it possible to set aside dedicated capacity for mission\sphinxhyphen{}critical applications that require low\sphinxhyphen{}latency communication—but also because the higher\sphinxhyphen{}frequency enables even finer\sphinxhyphen{}grain resource blocks (e.g., scheduling intervals as short as 0.125 ms). Again, this improves scheduling granularity to the benefit of applications that cannot tolerate unpredictable latency. \sphinxAtStartPar The fourth is related to delivering mobile connectivity to a massive number of IoT devices, ranging from devices that require mobility support and modest data rates (e.g. wearables, asset trackers) to devices that support intermittent transmission of a few bytes of data (e.g., sensors, meters). None of these devices are particularly latency\sphinxhyphen{}sensitive or bandwidth\sphinxhyphen{}hungry, but the latter are especially challenging because they require long battery lifetimes, and hence, reduced hardware complexity that draws less power. \sphinxAtStartPar Support for IoT device connectivity revolves around allocating some of the available radio spectrum to a light\sphinxhyphen{}weight (simplified) air interface. This approach started with Release 13 of LTE via two complementary technologies: mMTC and NB\sphinxhyphen{}IoT (NarrowBand\sphinxhyphen{}IoT). Both technologies build on a significantly simplified version of LTE—i.e., limiting the numerology and flexibility needed achieve high spectrum utilization—so as to allow for simpler IoT hardware design. mMTC delivers up to 1 Mbps over a 1.4 MHz of bandwidth and NB\sphinxhyphen{}IoT delivers a few tens of kbps over 200 kHz of bandwidth; hence the term \sphinxstyleemphasis{NarrowBand}. Both technologies have been designed to support over 1 million devices per square kilometer. With Release 16, both technologies can be operated in\sphinxhyphen{}band with 5G, but still based on LTE numerology. Starting with Release 17, a simpler version of 5G NR, called \sphinxstyleemphasis{NR\sphinxhyphen{}Light}, will be introduced as the evolution of mMTC. NR\sphinxhyphen{}Light is expected to scale the device density even further. \sphinxAtStartPar As a consequence of all four improvements, 5G NR is designed to support partitioning the available bandwidth, with different partitions dynamically allocated to different classes of traffic (e.g., high\sphinxhyphen{}bandwidth, low\sphinxhyphen{}latency, and low\sphinxhyphen{}complexity). This is the essence of \sphinxstyleemphasis{slicing}, an idea we will revisit throughout this book. Moreover, once traffic with different requirements can be served by different slices, 5G NR’s approach to multiplexing is general enough to support varied scheduling decisions for those slices, each tailored for the target traffic. \chapter{Chapter 3: Basic Architecture} \label{\detokenize{arch:chapter-3-basic-architecture}}\label{\detokenize{arch::doc}} \sphinxAtStartPar This chapter identifies the main architectural components of cellular access networks. It focuses on the components that are common to both 4G and 5G and, as such, establishes a foundation for understanding the advanced features of 5G presented in later chapters. \sphinxAtStartPar This overview is partly an exercise in introducing 3GPP terminology. For someone that is familiar with the Internet, this terminology can seem arbitrary (e.g., “eNB” is a “base station”), but it is important to keep in mind that this terminology came out of the 3GPP standardization process, which has historically been concerned about telephony and almost completely disconnected from the IETF and other Internet\sphinxhyphen{}related efforts. To further confuse matters, the 3GPP terminology often changes with each generation (e.g., a base station is called eNB in 4G and gNB in 5G). We address situations like this by using generic terminology (e.g., base station), and referencing the 3GPP\sphinxhyphen{}specific counterpart only when the distinction is helpful. \phantomsection\label{\detokenize{arch:reading-terminology}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar This example is only the tip of the terminology iceberg. For a slightly broader perspective on the complexity of terminology in 5G, see Marcin Dryjanski’s blog post: \sphinxhref{https://www.grandmetric.com/blog/2018/07/14/lte-and-5g-differences-system-complexity/}{LTE and 5G Differences: System Complexity}. July 2018. \end{sphinxadmonition} \section{3.1 Main Components} \label{\detokenize{arch:main-components}} \sphinxAtStartPar The cellular network provides wireless connectivity to devices that are on the move. These devices, which are known as \sphinxstyleemphasis{User Equipment (UE)}, have traditionally corresponded to smartphones and tablets, but will increasingly include cars, drones, industrial and agricultural machines, robots, home appliances, medical devices, and so on. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide01}.png} \caption{Cellular networks consists of a Radio Access Network (RAN) and a Mobile Core.}\label{\detokenize{arch:id2}}\label{\detokenize{arch:fig-cellular}}\end{figure} \sphinxAtStartPar As shown in \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}, the cellular network consists of two main subsystems: the \sphinxstyleemphasis{Radio Access Network (RAN)} and the \sphinxstyleemphasis{Mobile Core}. The RAN manages the radio spectrum, making sure it is used efficiently and meets the quality\sphinxhyphen{}of\sphinxhyphen{}service requirements of every user. It corresponds to a distributed collection of base stations. As noted above, in 4G these are (somewhat cryptically) named \sphinxstyleemphasis{eNodeB} (or \sphinxstyleemphasis{eNB}), which is short for \sphinxstyleemphasis{evolved Node B}. In 5G they are known as \sphinxstyleemphasis{gNB}. (The g stands for “next Generation”.) \sphinxAtStartPar The Mobile Core is a bundle of functionality (as opposed to a device) that serves several purposes. \begin{itemize} \item {} \sphinxAtStartPar Provides Internet (IP) connectivity for both data and voice services. \item {} \sphinxAtStartPar Ensures this connectivity fulfills the promised QoS requirements. \item {} \sphinxAtStartPar Tracks user mobility to ensure uninterrupted service. \item {} \sphinxAtStartPar Tracks subscriber usage for billing and charging. \end{itemize} \sphinxAtStartPar Note that Mobile Core is another example of a generic term. In 4G this is called the \sphinxstyleemphasis{Evolved Packet Core (EPC)} and in 5G it is called the \sphinxstyleemphasis{Next Generation Core (NG\sphinxhyphen{}Core)}. \sphinxAtStartPar Even though the word “Core” is in its name, from an Internet perspective, the Mobile Core is still part of the access network, effectively providing a bridge between the RAN in some geographic area and the greater IP\sphinxhyphen{}based Internet. 3GPP provides significant flexibility in how the Mobile Core is geographically deployed, but for our purposes, assuming each instantiation of the Mobile Core serves a metropolitan area is a good working model. The corresponding RAN would then span several dozens (or even hundreds) of cell towers. \sphinxAtStartPar Taking a closer look at \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}, we see that a \sphinxstyleemphasis{Backhaul Network} interconnects the base stations that implement the RAN with the Mobile Core. This network is typically wired, may or may not have the ring topology shown in the Figure, and is often constructed from commodity components found elsewhere in the Internet. For example, the \sphinxstyleemphasis{Passive Optical Network (PON)} that implements Fiber\sphinxhyphen{}to\sphinxhyphen{}the\sphinxhyphen{}Home is a prime candidate for implementing the RAN backhaul. The backhaul network is obviously a necessary part of the RAN, but it is an implementation choice and not prescribed by the 3GPP standard. \sphinxAtStartPar Although 3GPP specifies all the elements that implement the RAN and Mobile Core in an open standard—including sub\sphinxhyphen{}layers we have not yet introduced—network operators have historically bought proprietary implementations of each subsystem from a single vendor. This lack of an open source implementation contributes to the perceived “opaqueness” of the cellular network in general, and the RAN in particular. And while it is true that an eNodeB implementation does contain sophisticated algorithms for scheduling transmission on the radio spectrum—algorithms that are considered valuable intellectual property of the equipment vendors—there is significant opportunity to open and disaggregate both the RAN and the Mobile Core. The following two sections describe each, in turn. \sphinxAtStartPar Before getting to those details, \hyperref[\detokenize{arch:fig-cups}]{Figure \ref{\detokenize{arch:fig-cups}}} redraws components from \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}} to highlight two important distinctions. The first is that a base station has an analog component (depicted by an antenna) and a digital component (depicted by a processor pair). The second is that the Mobile Core is partitioned into a \sphinxstyleemphasis{Control Plane} and \sphinxstyleemphasis{User Plane}, which is similar to the control/data plane split that someone familiar with the Internet would recognize. (3GPP also recently introduced a corresponding acronym—\sphinxstyleemphasis{CUPS, Control and User Plane Separation}—to denote this idea.) The importance of these two distinctions will become clear in the following discussion. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide02}.png} \caption{Mobile Core divided into a Control Plan and a User Plane, an architectural feature known as CUPS: Control and User Plane Separation}\label{\detokenize{arch:id3}}\label{\detokenize{arch:fig-cups}}\end{figure} \section{3.2 Radio Access Network} \label{\detokenize{arch:radio-access-network}} \sphinxAtStartPar We now describe the RAN by sketching the role each base station plays. Keep in mind this is kind of like describing the Internet by explaining how a router works—a not unreasonable place to start, but it doesn’t fully do justice to the end\sphinxhyphen{}to\sphinxhyphen{}end story. \sphinxAtStartPar First, each base station establishes the wireless channel for a subscriber’s UE upon power\sphinxhyphen{}up or upon handover when the UE is active. This channel is released when the UE remains idle for a predetermined period of time. Using 3GPP terminology, this wireless channel is said to provide a \sphinxstyleemphasis{bearer service}. The term “bearer” has historically been used in telecommunications (including early wireline technologies like ISDN) to denote a data channel, as opposed to a channel that carries signaling information. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide03}.png} \caption{Base Station detects (and connects to) active UEs.}\label{\detokenize{arch:id4}}\label{\detokenize{arch:fig-active-ue}}\end{figure} \sphinxAtStartPar Second, each base station establishes “3GPP Control Plane” connectivity between the UE and the corresponding Mobile Core Control Plane component, and forwards signaling traffic between the two. This signaling traffic enables UE authentication, registration, and mobility tracking. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide04}.png} \caption{Base Station establishes control plane connectivity between each UE and the Mobile Core.}\label{\detokenize{arch:id5}}\label{\detokenize{arch:fig-control-plane}}\end{figure} \sphinxAtStartPar Third, for each active UE, the base station establishes one or more tunnels between the corresponding Mobile Core User Plane component. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide05}.png} \caption{Base station establishes one or more tunnels between each UE and the Mobile Core’s User Plane.}\label{\detokenize{arch:id6}}\label{\detokenize{arch:fig-user-plane}}\end{figure} \sphinxAtStartPar Fourth, the base station forwards both control and user plane packets between the Mobile Core and the UE. These packets are tunnelled over SCTP/IP and GTP/UDP/IP, respectively. SCTP (Stream Control Transport Protocol) is an alternative reliable transport to TCP, tailored to carry signaling (control) information for telephony services. GTP (a nested acronym corresponding to (General Packet Radio Service) Tunneling Protocol) is a 3GPP\sphinxhyphen{}specific tunneling protocol designed to run over UDP. \sphinxAtStartPar As an aside, it is noteworthy that connectivity between the RAN and the Mobile Core is IP\sphinxhyphen{}based. This was introduced as one of the main changes between 3G and 4G. Prior to 4G, the internals of the cellular network were circuit\sphinxhyphen{}based, which is not surprising given its origins as a voice network. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide06}.png} \caption{Base Station to Mobile Core (and Base Station to Base Station) control plane tunneled over SCTP/IP and user plane tunneled over GTP/UDP/IP.}\label{\detokenize{arch:id7}}\label{\detokenize{arch:fig-tunnels}}\end{figure} \sphinxAtStartPar Fifth, each base station coordinates UE handovers with neighboring base stations, using direct station\sphinxhyphen{}to\sphinxhyphen{}station links. Exactly like the station\sphinxhyphen{}to\sphinxhyphen{}core connectivity shown in the previous figure, these links are used to transfer both control plane (SCTP over IP) and user plane (GTP over UDP/IP) packets. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide07}.png} \caption{Base Stations cooperate to implement UE hand over.}\label{\detokenize{arch:id8}}\label{\detokenize{arch:fig-handover}}\end{figure} \sphinxAtStartPar Sixth, the base stations coordinate wireless multi\sphinxhyphen{}point transmission to a UE from multiple base stations, which may or may not be part of a UE handover from one base station to another. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide08}.png} \caption{Base Stations cooperate to implement multipath transmission (link aggregation) to UEs.}\label{\detokenize{arch:id9}}\label{\detokenize{arch:fig-link-aggregation}}\end{figure} \sphinxAtStartPar The main takeaway is that the base station can be viewed as a specialized forwarder. In the Internet\sphinxhyphen{}to\sphinxhyphen{}UE direction, it fragments outgoing IP packets into physical layer segments and schedules them for transmission over the available radio spectrum, and in the UE\sphinxhyphen{}to\sphinxhyphen{}Internet direction it assembles physical layer segments into IP packets and forwards them (over a GTP/UDP/IP tunnel) to the upstream user plane of the Mobile Core. Also, based on observations of the wireless channel quality and per\sphinxhyphen{}subscriber policies, it decides whether to (a) forward outgoing packets directly to the UE, (b) indirectly forward packets to the UE via a neighboring base station, or (c) utilize multiple paths to reach the UE. The third case has the option of either spreading the physical payloads across multiple base stations or across multiple carrier frequencies of a single base station (including Wi\sphinxhyphen{}Fi). \sphinxAtStartPar Note that as outlined in Chapter 2, scheduling is complex and multi\sphinxhyphen{}faceted, even when viewed as a localized decision at a single base station. What we now see is that there is also a global element, whereby it’s possible to forward traffic to a different base station (or to multiple base stations) in an effort to make efficient use of the radio spectrum over a larger geographic area. \sphinxAtStartPar In other words, the RAN as a whole (i.e., not just a single base station) not only supports handovers (an obvious requirement for mobility), but also \sphinxstyleemphasis{link aggregation} and \sphinxstyleemphasis{load balancing}, mechanisms that are familiar to anyone who understands the Internet. We will revisit how such RAN\sphinxhyphen{}wide (global) decisions can be made using SDN techniques in a later chapter. \section{3.3 Mobile Core} \label{\detokenize{arch:mobile-core}} \sphinxAtStartPar The main function of the Mobile Core is to provide external packet data network (i.e., Internet) connectivity to mobile subscribers, while ensuring that they are authenticated and their observed service qualities satisfy their subscription SLAs. An important aspect of the Mobile Core is that it needs to manage all subscribers’ mobility by keeping track of their last whereabouts at the granularity of the serving base station. It’s the fact that the Mobile Core is keeping track of individual subscribers—something that the Internet’s core does not do—that creates a lot of the complexity in its architecture, especially given that those subscribers are moving around. \sphinxAtStartPar While the aggregate functionality remains largely the same as we migrate from 4G to 5G, how that functionality is virtualized and factored into individual components changes. The 5G Mobile Core is heavily influenced by the cloud’s march toward a microservice\sphinxhyphen{}based (cloud native) architecture. This shift to cloud native is deeper than it might first appear, in part because it opens the door to customization and specialization. Instead of supporting just voice and broadband connectivity, the 5G Mobile Core can evolve to also support, for example, massive IoT, which has a fundamentally different latency requirement and usage pattern (i.e., many more devices connecting intermittently). This stresses—if not breaks—a one\sphinxhyphen{}size\sphinxhyphen{}fits\sphinxhyphen{}all approach to session management. \subsection{4G Mobile Core} \label{\detokenize{arch:g-mobile-core}} \sphinxAtStartPar The 4G Mobile Core, which 3GPP officially refers to as the \sphinxstyleemphasis{Evolved Packet Core (EPC)}, consists of five main components, the first three of which run in the Control Plane (CP) and the second two of which run in the User Plane (UP). \begin{itemize} \item {} \sphinxAtStartPar MME (Mobility Management Entity): Tracks and manages the movement of UEs throughout the RAN. This includes recording when the UE is not active. \item {} \sphinxAtStartPar HSS (Home Subscriber Server): A database that contains all subscriber\sphinxhyphen{}related information. \item {} \sphinxAtStartPar PCRF (Policy \& Charging Rules Function): Tracks and manages policy rules and records billing data on subscriber traffic. \item {} \sphinxAtStartPar SGW (Serving Gateway): Forwards IP packets to and from the RAN. Anchors the Mobile Core end of the bearer service to a (potentially mobile) UE, and so is involved in handovers from one base station to another. \item {} \sphinxAtStartPar PGW (Packet Gateway): Essentially an IP router, connecting the Mobile Core to the external Internet. Supports additional access\sphinxhyphen{}related functions, including policy enforcement, traffic shaping, and charging. \end{itemize} \sphinxAtStartPar Although specified as distinct components, in practice the SGW (RAN\sphinxhyphen{}facing) and PGW (Internet\sphinxhyphen{}facing) are often combined in a single device, commonly referred to as an S/PGW. The end result is illustrated in \hyperref[\detokenize{arch:fig-4g-core}]{Figure \ref{\detokenize{arch:fig-4g-core}}}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide20}.png} \caption{4G Mobile Core (Evolved Packet Core).}\label{\detokenize{arch:id10}}\label{\detokenize{arch:fig-4g-core}}\end{figure} \sphinxAtStartPar Note that 3GPP is flexible in how the Mobile Core components are deployed to serve a geographic area. For example, a single MME/PGW pair might serve a metropolitan area, with SGWs deployed across \textasciitilde{}10 edge sites spread throughout the city, each of which serves \textasciitilde{}100 base stations. But alternative deployment configurations are allowed by the spec. \subsection{5G Mobile Core} \label{\detokenize{arch:id1}} \sphinxAtStartPar The 5G Mobile Core, which 3GPP calls the \sphinxstyleemphasis{NG\sphinxhyphen{}Core}, adopts a microservice\sphinxhyphen{}like architecture, where we say “microservice\sphinxhyphen{}like” because while the 3GPP specification spells out this level of disaggregation, it is really just prescribing a set of functional blocks and not an implementation. A set of functional blocks is very different from the collection of engineering decisions that go into designing a microservice\sphinxhyphen{}based system. That said, viewing the collection of components shown in \hyperref[\detokenize{arch:fig-5g-core}]{Figure \ref{\detokenize{arch:fig-5g-core}}} as a set of microservices is a good working model. \sphinxAtStartPar The following organizes the set of functional blocks into three groups. The first group runs in the Control Plane (CP) and has a counterpart in the EPC. \begin{itemize} \item {} \sphinxAtStartPar AMF (Core Access and Mobility Management Function): Responsible for connection and reachability management, mobility management, access authentication and authorization, and location services. Manages the mobility\sphinxhyphen{}related aspects of the EPC’s MME. \item {} \sphinxAtStartPar SMF (Session Management Function): Manages each UE session, including IP address allocation, selection of associated UP function, control aspects of QoS, and control aspects of UP routing. Roughly corresponds to part of the EPC’s MME and the control\sphinxhyphen{}related aspects of the EPC’s PGW. \item {} \sphinxAtStartPar PCF (Policy Control Function): Manages the policy rules that other CP functions then enforce. Roughly corresponds to the EPC’s PCRF. \item {} \sphinxAtStartPar UDM (Unified Data Management): Manages user identity, including the generation of authentication credentials. Includes part of the functionality in the EPC’s HSS. \item {} \sphinxAtStartPar AUSF (Authentication Server Function): Essentially an authentication server. Includes part of the functionality in the EPC’s HSS. \end{itemize} \sphinxAtStartPar The second group also runs in the Control Plane (CP) but does not have a direct counterpart in the EPC: \begin{itemize} \item {} \sphinxAtStartPar SDSF (Structured Data Storage Network Function): A “helper” service used to store structured data. Could be implemented by an “SQL Database” in a microservices\sphinxhyphen{}based system. \item {} \sphinxAtStartPar UDSF (Unstructured Data Storage Network Function): A “helper” service used to store unstructured data. Could be implemented by a “Key/Value Store” in a microservices\sphinxhyphen{}based system. \item {} \sphinxAtStartPar NEF (Network Exposure Function): A means to expose select capabilities to third\sphinxhyphen{}party services, including translation between internal and external representations for data. Could be implemented by an “API Server” in a microservices\sphinxhyphen{}based system. \item {} \sphinxAtStartPar NRF (NF Repository Function): A means to discover available services. Could be implemented by a “Discovery Service” in a microservices\sphinxhyphen{}based system. \item {} \sphinxAtStartPar NSSF (Network Slicing Selector Function): A means to select a Network Slice to serve a given UE. Network slices are essentially a way to partition network resources in order to differentiate service given to different users. It is a key feature of 5G that we discuss in depth in a later chapter. \end{itemize} \sphinxAtStartPar The third group includes the one component that runs in the User Plane (UP): \begin{itemize} \item {} \sphinxAtStartPar UPF (User Plane Function): Forwards traffic between RAN and the Internet, corresponding to the S/PGW combination in EPC. In addition to packet forwarding, it is responsible for policy enforcement, lawful intercept, traffic usage reporting, and QoS policing. \end{itemize} \sphinxAtStartPar Of these, the first and third groups are best viewed as a straightforward refactoring of 4G’s EPC, while the second group—despite the gratuitous introduction of new terminology—is 3GPP’s way of pointing to a cloud native solution as the desired end\sphinxhyphen{}state for the Mobile Core. Of particular note, introducing distinct storage services means that all the other services can be stateless, and hence, more readily scalable. Also note that \hyperref[\detokenize{arch:fig-5g-core}]{Figure \ref{\detokenize{arch:fig-5g-core}}} adopts an idea that’s common in microservice\sphinxhyphen{}based systems, namely, to show a \sphinxstyleemphasis{message bus} interconnecting all the components rather than including a full set of pairwise connections. This also suggests a well\sphinxhyphen{}understood implementation strategy. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide33}.png} \caption{5G Mobile Core (NG\sphinxhyphen{}Core).}\label{\detokenize{arch:id11}}\label{\detokenize{arch:fig-5g-core}}\end{figure} \sphinxAtStartPar Stepping back from these details, and with the caveat that we are presuming an implementation, the main takeaway is that we can conceptualize the Mobile Core as a graph of services. You will sometimes hear this called a \sphinxstyleemphasis{Service Graph} or \sphinxstyleemphasis{Service Chain}, the latter being more prevalent in NFV\sphinxhyphen{}oriented documents. Another term, \sphinxstyleemphasis{Service Mesh}, has taken on a rather specific meaning in cloud native terminology—we’ll avoid overloading that term here. 3GPP is silent on the specific terminology since it is considered an implementation choice rather than part of the specification. We describe our implementation choices in later chapters. \section{3.4 Security and Mobility} \label{\detokenize{arch:security-and-mobility}} \sphinxAtStartPar We now take a closer look at two unique features of the cellular network—its support for security and mobility—both of which differentiate it from WiFi. The following also serves to fill in some details about how each individual UE connects to the network. \sphinxAtStartPar We start with the security architecture, which is grounded in two trust assumptions. First, each Base Station trusts that it is connected to the Mobile Core by a secure private network, over which it establishes the tunnels introduced in \hyperref[\detokenize{arch:fig-tunnels}]{Figure \ref{\detokenize{arch:fig-tunnels}}}: a GTP/UDP/IP tunnel to the Core’s User Plane (Core\sphinxhyphen{}UP) and a SCTP/IP tunnel to the Core’s Control Plane (Core\sphinxhyphen{}CP). Second, each UE has an operator\sphinxhyphen{}provided SIM card, which uniquely identifies the subscriber (i.e., phone number) and establishes the radio parameters (e.g., frequency band) needed to communicate with that operator’s Base Stations. The SIM card also includes a secret key that the UE uses to authenticate itself. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide34}.png} \caption{Sequence of steps to establish secure Control and User Plane channels.}\label{\detokenize{arch:id12}}\label{\detokenize{arch:fig-secure}}\end{figure} \sphinxAtStartPar With this starting point, \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} shows the per\sphinxhyphen{}UE connection sequence. When a UE first becomes active, it communicates with a nearby Base Station over a temporary (unauthenticated) radio link (Step 1). The Base Station forwards the request to the Core\sphinxhyphen{}CP over the existing tunnel, and the Core\sphinxhyphen{}CP (specifically, the MME in 4G and the AMF in 5G) initiates an authentication protocol with the UE (Step 2). 3GPP identifies a set of options for authentication and encryption, where the actual protocols used are an implementation choice. For example, \sphinxstyleemphasis{Advanced Encryption Standard} (AES) is one of the options for encryption. Note that this authentication exchange is initially in the clear since the Base Station to UE link is not yet secure. \sphinxAtStartPar Once the UE and Core\sphinxhyphen{}CP are satisfied with each other’s identity, the Core\sphinxhyphen{}CP informs the other components of the parameters they will need to service the UE (Step 3). This includes: (a) instructing the Core\sphinxhyphen{}UP to initialize the user plane (e.g., assign an IP address to the UE and set the appropriate QCI parameter); (b) instructing the Base Station to establish an encrypted channel to the UE; and (c) giving the UE the symmetric key it will need to use the encrypted channel with the Base Station. The symmetric key is encrypted using the public key of the UE (so only the UE can decrypt it, using its secret key). Once complete, the UE can use the end\sphinxhyphen{}to\sphinxhyphen{}end user plane channel through the Core\sphinxhyphen{}UP (Step 4). \sphinxAtStartPar There are three additional details of note about this process. First, the secure control channel between the UE and the Core\sphinxhyphen{}CP set up during Step 2 remains available, and is used by the Core\sphinxhyphen{}CP to send additional control instructions to the UE during the course of the session. \sphinxAtStartPar Second, the user plane channel established during Step 4 is referred to as the \sphinxstyleemphasis{Default Bearer Service}, but additional channels can be established between the UE and Core\sphinxhyphen{}UP, each with a potentially different QCI value. This might be done on an application\sphinxhyphen{}by\sphinxhyphen{}application basis, for example, under the control of the Mobile Core doing \sphinxstyleemphasis{Deep Packet Inspection} (DPI) on the traffic, looking for flows that require special treatment. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide35}.png} \caption{Sequence of per\sphinxhyphen{}hop tunnels involved in an end\sphinxhyphen{}to\sphinxhyphen{}end User Plane channel.}\label{\detokenize{arch:id13}}\label{\detokenize{arch:fig-per-hop}}\end{figure} \sphinxAtStartPar Third, while the resulting user plane channels are logically end\sphinxhyphen{}to\sphinxhyphen{}end, each is actually implemented as a sequence of per\sphinxhyphen{}hop tunnels, as illustrated in \hyperref[\detokenize{arch:fig-per-hop}]{Figure \ref{\detokenize{arch:fig-per-hop}}}. (The figure shows the SGW and PGW from the 4G Mobile Core to make the example more concrete.) This means each component on the end\sphinxhyphen{}to\sphinxhyphen{}end path terminates a downstream tunnel using one local identifier for a given UE, and initiates an upstream tunnel using a second local identifier for that UE. In practice, these per\sphinxhyphen{}flow tunnels are often bundled into an single inter\sphinxhyphen{}component tunnel, which makes it impossible to differentiate the level of service given to any particular end\sphinxhyphen{}to\sphinxhyphen{}end UE channel. This is a limitation of 4G that 5G has ambitions to correct. \sphinxAtStartPar Support for mobility can now be understood as the process of re\sphinxhyphen{}executing one or more of the steps shown in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} as the UE moves throughout the RAN. The unauthenticated link indicated by (1) allows the UE to be known to all Base Station within range. (We refer to these as \sphinxstyleemphasis{potential links} in later chapters.) Based on the signal’s measured CQI, the Base Stations communicate directly with each other to make a handover decision. Once made, the decision is then communicated to the Mobile Core, re\sphinxhyphen{}triggering the setup functions indicated by (3), which in turn re\sphinxhyphen{}builds the user plane tunnel between the Base Station and the SGW shown in \hyperref[\detokenize{arch:fig-per-hop}]{Figure \ref{\detokenize{arch:fig-per-hop}}} (or correspondingly, between the Base Station and the UPF in 5G). One of the most unique features of the cellular network is that the Mobile Core’s user plane (e.g., UPF in 5G) buffers data during the handover transition, avoiding dropped packets and subsequent end\sphinxhyphen{}to\sphinxhyphen{}end retransmissions. \sphinxAtStartPar In other words, the cellular network maintains the \sphinxstyleemphasis{UE session} in the face of mobility (corresponding to the control and data channels depicted by (2) and (4) in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}}, respectively), but it is able to do so only when the same Mobile Core serves the UE (i.e., only the Base Station changes). This would typically be the case for a UE moving within a metropolitan area. Moving between metro areas—and hence, between Mobile Cores—is indistinguishable from power cycling a UE. The UE is assigned a new IP address and no attempt is made to buffer and subsequently deliver in\sphinxhyphen{}flight data. Independent of mobility, but relevant to this discussion, any UE that becomes inactive for a period of time also loses its session, with a new session established and a new IP address assigned when the UE becomes active again. \sphinxAtStartPar Note that this session\sphinxhyphen{}based approach can be traced to the cellular network’s roots as a connection\sphinxhyphen{}oriented network. An interesting thought experiment is whether the Mobile Core will continue to evolve so as to better match the connectionless assumptions of the Internet protocols that typically run on top of it. \section{3.5 Deployment Options} \label{\detokenize{arch:deployment-options}} \sphinxAtStartPar With an already deployed 4G RAN/EPC in the field and a new 5G RAN/NG\sphinxhyphen{}Core deployment underway, we can’t ignore the issue of transitioning from 4G to 5G (an issue the IP\sphinxhyphen{}world has been grappling with for 20 years). 3GPP officially spells out multiple deployment options, which can be summarized as follows. \begin{itemize} \item {} \sphinxAtStartPar Standalone 4G / Stand\sphinxhyphen{}Alone 5G \item {} \sphinxAtStartPar Non\sphinxhyphen{}Standalone (4G+5G RAN) over 4G’s EPC \item {} \sphinxAtStartPar Non\sphinxhyphen{}Standalone (4G+5G RAN) over 5G’s NG\sphinxhyphen{}Core \end{itemize} \sphinxAtStartPar The second of the three options, which is generally referred to as “NSA“, involves 5G base stations being deployed alongside the existing 4G base stations in a given geography to provide a data\sphinxhyphen{}rate and capacity boost. In NSA, control plane traffic between the user equipment and the 4G Mobile Core utilizes (i.e., is forwarded through) 4G base stations, and the 5G base stations are used only to carry user traffic. Eventually, it is expected that operators complete their migration to 5G by deploying NG Core and connecting their 5G base stations to it for Standalone (SA) operation. NSA and SA operations are illustrated in \hyperref[\detokenize{arch:fig-nsa}]{Figure \ref{\detokenize{arch:fig-nsa}}}. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide38}.png} \caption{NSA and SA options for 5G deployment.}\label{\detokenize{arch:id14}}\label{\detokenize{arch:fig-nsa}}\end{figure} \sphinxAtStartPar One reason we call attention to the phasing issue is that we face a similar challenge in the chapters that follow. The closer the following discussion gets to implementation details, the more specific we have to be about whether we are using 4G components or 5G components. As a general rule, we use 4G components—particularly with respect to the Mobile Core, since that’s what’s available in open source today—and trust the reader can make the appropriate substitution without loss of generality. Like the broader industry, the open source community is in the process of incrementally evolving its 4G code base into its 5G\sphinxhyphen{}compliant counterpart. \phantomsection\label{\detokenize{arch:reading-migration}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For more insight into 4G to 5G migration strategies, see \sphinxhref{https://www.gsma.com/futurenetworks/wp-content/uploads/2018/04/Road-to-5G-Introduction-and-Migration\_FINAL.pdf}{Road to 5G: Introduction and Migration}. GSMA Report, April 2018. \end{sphinxadmonition} \chapter{Chapter 4: RAN Internals} \label{\detokenize{ran:chapter-4-ran-internals}}\label{\detokenize{ran::doc}} \sphinxAtStartPar The description of the RAN in the previous chapter focused on functionality, but was mostly silent about the RAN’s internal structure. We now focus on some of the internal details, and in doing so, explain how the RAN is being transformed in 5G. This involves first describing the stages in the packet processing pipeline, and then showing how these stages can be disaggregated, distributed and implemented. \sphinxAtStartPar Our approach in this chapter is to incrementally build the RAN from the bottom up in the first three sections. Section 4.4 then summarizes the overall design, with a focus on how the resulting end\sphinxhyphen{}to\sphinxhyphen{}end system is architected to evolve. \section{4.1 Packet Processing Pipeline} \label{\detokenize{ran:packet-processing-pipeline}} \sphinxAtStartPar \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} shows the packet processing stages implemented by the base station. These stages are specified by the 3GPP standard. Note that the figure depicts the base station as a pipeline (running left\sphinxhyphen{}to\sphinxhyphen{}right for packets sent to the UE) but it is equally valid to view it as a protocol stack (as is typically done in official 3GPP documents). Also note that (for now) we are agnostic as to how these stages are implemented, but since we are ultimately heading towards a cloud\sphinxhyphen{}based implementation, you can think of each as corresponding to a microservice (if that is helpful). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide14}.png} \caption{RAN processing pipeline, including both user and control plane components.}\label{\detokenize{ran:id1}}\label{\detokenize{ran:fig-pipeline}}\end{figure} \sphinxAtStartPar The key stages are as follows. \begin{itemize} \item {} \sphinxAtStartPar RRC (Radio Resource Control) \(\rightarrow\) Responsible for configuring the coarse\sphinxhyphen{}grain and policy\sphinxhyphen{}related aspects of the pipeline. The RRC runs in the RAN’s control plane; it does not process packets on the user plane. \item {} \sphinxAtStartPar PDCP (Packet Data Convergence Protocol) \(\rightarrow\) Responsible for compressing and decompressing IP headers, ciphering and integrity protection, and making an “early” forwarding decision (i.e., whether to send the packet down the pipeline to the UE or forward it to another base station). \item {} \sphinxAtStartPar RLC (Radio Link Control) \(\rightarrow\) Responsible for segmentation and reassembly, including reliably transmitting/receiving segments by implementing a form of ARQ (automatic repeat request). \item {} \sphinxAtStartPar MAC (Media Access Control) \(\rightarrow\) Responsible for buffering, multiplexing and demultiplexing segments, including all real\sphinxhyphen{}time scheduling decisions about what segments are transmitted when. Also able to make a “late” forwarding decision (i.e., to alternative carrier frequencies, including Wi\sphinxhyphen{}Fi). \item {} \sphinxAtStartPar PHY (Physical Layer) \(\rightarrow\) Responsible for coding and modulation (as discussed in an earlier chapter), including FEC. \end{itemize} \sphinxAtStartPar The last two stages in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} (D/A conversion and the RF front\sphinxhyphen{}end) are beyond the scope of this book. \sphinxAtStartPar While it is simplest to view the stages in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} as a pure left\sphinxhyphen{}to\sphinxhyphen{}right pipeline, in practice the Scheduler running in the MAC stage implements the “main loop” for outbound traffic, reading data from the upstream RLC and scheduling transmissions to the downstream PHY. In particular, since the Scheduler determines the number of bytes to transmit to a given UE during each time period (based on all the factors outlined in an earlier chapter), it must request (get) a segment of that length from the upstream queue. In practice, the size of the segment that can be transmitted on behalf of a single UE during a single scheduling interval can range from a few bytes to an entire IP packet. \section{4.2 Split RAN} \label{\detokenize{ran:split-ran}} \sphinxAtStartPar The next step is to understand how the functionality outlined above is partitioned between physical elements, and hence, “split” across centralized and distributed locations. The dominant option has historically been “no split,” with the entire pipeline shown in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} running in the base station. Going forward, the 3GPP standard has been extended to allow for multiple split\sphinxhyphen{}points, with the partition shown in \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}} being actively pursued by the operator\sphinxhyphen{}led O\sphinxhyphen{}RAN (Open RAN) Alliance. It is the split we adopt throughout the rest of this book. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide15}.png} \caption{Split\sphinxhyphen{}RAN processing pipeline distributed across a Central Unit (CU), Distributed Unit (DU), and Radio Unit (RU).}\label{\detokenize{ran:id2}}\label{\detokenize{ran:fig-split-ran}}\end{figure} \sphinxAtStartPar This results in a RAN\sphinxhyphen{}wide configuration similar to that shown in \hyperref[\detokenize{ran:fig-ran-hierarchy}]{Figure \ref{\detokenize{ran:fig-ran-hierarchy}}}, where a single \sphinxstyleemphasis{Central Unit (CU)} running in the cloud serves multiple \sphinxstyleemphasis{Distributed Units (DUs)}, each of which in turn serves multiple \sphinxstyleemphasis{Radio Units (RUs)}. Critically, the RRC (centralized in the CU) is responsible for only near\sphinxhyphen{}real\sphinxhyphen{}time configuration and control decision making, while the Scheduler that is part of the MAC stage is responsible for all real\sphinxhyphen{}time scheduling decisions. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide16}.png} \caption{Split\sphinxhyphen{}RAN hierarchy, with one CU serving multiple DUs, each of which serves multiple RUs.}\label{\detokenize{ran:id3}}\label{\detokenize{ran:fig-ran-hierarchy}}\end{figure} \sphinxAtStartPar Because scheduling decisions for radio transmission are made by the MAC layer in real time, a DU needs to be “near” (within 1ms) the RUs it manages. (You can’t afford to make scheduling decisions based on out\sphinxhyphen{}of\sphinxhyphen{}date channel information.) One familiar configuration is to co\sphinxhyphen{}locate a DU and an RU in a cell tower. But when an RU corresponds to a small cell, many of which might be spread across a modestly\sphinxhyphen{}sized geographic area (e.g., a mall, campus, or factory), then a single DU would likely service multiple RUs. The use of mmWave in 5G is likely to make this later configuration all the more common. \sphinxAtStartPar Also note that the split\sphinxhyphen{}RAN changes the nature of the Backhaul Network, which in 4G connected the base stations (eNBs) back to the Mobile Core. With the split\sphinxhyphen{}RAN there are multiple connections, which are officially labeled as follows. \begin{itemize} \item {} \sphinxAtStartPar RU\sphinxhyphen{}DU connectivity is called the Fronthaul \item {} \sphinxAtStartPar DU\sphinxhyphen{}CU connectivity is called the Midhaul \item {} \sphinxAtStartPar CU\sphinxhyphen{}Mobile Core connectivity is called the Backhaul \end{itemize} \sphinxAtStartPar One observation about the CU (which is relevant in the next chapter) is that one might co\sphinxhyphen{}locate the CU and Mobile Core in the same cluster, meaning the backhaul is implemented in the cluster switching fabric. In such a configuration, the midhaul then effectively serves the same purpose as the original backhaul, and the fronthaul is constrained by the predictable/low\sphinxhyphen{}latency requirements of the MAC stage’s real\sphinxhyphen{}time scheduler. \sphinxAtStartPar A second observation about the CU shown in \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}} is that it encompasses two functional blocks—the RRC and the PDCP—which lie on the RAN’s control plane and user plane, respectively. This separation is consistent with the idea of CUPS introduced in Chapter 3, and plays an increasingly important role as we dig deeper into how the RAN is implemented. For now, we note that the two parts are typically referred to as the CU\sphinxhyphen{}C and CU\sphinxhyphen{}U, respectively. \phantomsection\label{\detokenize{ran:reading-backhaul}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For more insight into design considerations for interconnecting the distributed components of a Split RAN, see \sphinxhref{https://www.ngmn.org/wp-content/uploads/NGMN\_RANEV\_D4\_BH\_FH\_Evolution\_V1.01.pdf}{RAN Evolution Project: Backhaul and Fronthaul Evolution}. NGMN Alliance Report, March 2015. \end{sphinxadmonition} \section{4.3 Software\sphinxhyphen{}Defined RAN} \label{\detokenize{ran:software-defined-ran}} \sphinxAtStartPar We now describe how the RAN is implemented according to SDN principles, resulting in an SD\sphinxhyphen{}RAN. The key architectural insight is shown in \hyperref[\detokenize{ran:fig-rrc-split}]{Figure \ref{\detokenize{ran:fig-rrc-split}}}, where the RRC from \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} is partitioned into two sub\sphinxhyphen{}components: the one on the left provides a 3GPP\sphinxhyphen{}compliant way for the RAN to interface to the Mobile Core’s control plane, while the one on the right opens a new programmatic API for exerting software\sphinxhyphen{}based control over the pipeline that implements the RAN user plane. \sphinxAtStartPar To be more specific, the left sub\sphinxhyphen{}component simply forwards control packets between the Mobile Core and the PDCP, providing a path over which the Mobile Core can communicate with the UE for control purposes, whereas the right sub\sphinxhyphen{}component implements the core of the RRC’s control functionality. This component is commonly referred to as the \sphinxstyleemphasis{RAN Intelligent Controller (RIC)} in O\sphinxhyphen{}RAN architecture documents, so we adopt this terminology. The “Near\sphinxhyphen{}Real Time” qualifier indicates the RIC is part of 10\sphinxhyphen{}100ms control loop implemented in the CU, as opposed to the \textasciitilde{}1ms control loop required by the MAC scheduler running in the DU. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide18}.png} \caption{RRC disaggregated into a Mobile Core facing control plane component and a Near\sphinxhyphen{}Real\sphinxhyphen{}Time Controller.}\label{\detokenize{ran:id4}}\label{\detokenize{ran:fig-rrc-split}}\end{figure} \sphinxAtStartPar Although not shown in \hyperref[\detokenize{ran:fig-rrc-split}]{Figure \ref{\detokenize{ran:fig-rrc-split}}}, keep in mind (from \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}}) that all constituent parts of the RRC, plus the PDCP, form the CU. \sphinxAtStartPar Completing the picture, \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}} shows the Near\sphinxhyphen{}RT RIC implemented as an SDN Controller hosting a set of SDN control apps. The RIC maintains a \sphinxstyleemphasis{RAN Network Information Base (R\sphinxhyphen{}NIB)}\textendash{}a common set of information that can be consumed by numerous control apps. The R\sphinxhyphen{}NIB includes time\sphinxhyphen{}averaged CQI values and other per\sphinxhyphen{}session state (e.g., GTP tunnel IDs, QCI values for the type of traffic), while the MAC (as part of the DU) maintains the instantaneous CQI values required by the real\sphinxhyphen{}time scheduler. Specifically, the R\sphinxhyphen{}NIB includes the following state. \begin{itemize} \item {} \sphinxAtStartPar NODES: Base Stations and Mobile Devices \begin{itemize} \item {} \sphinxAtStartPar Base Station Attributes: \begin{itemize} \item {} \sphinxAtStartPar Identifiers \item {} \sphinxAtStartPar Version \item {} \sphinxAtStartPar Config Report \item {} \sphinxAtStartPar RRM config \item {} \sphinxAtStartPar PHY resource usage \end{itemize} \item {} \sphinxAtStartPar Mobile Device Attributes: \begin{itemize} \item {} \sphinxAtStartPar Identifiers \item {} \sphinxAtStartPar Capability \item {} \sphinxAtStartPar Measurement Config \item {} \sphinxAtStartPar State (Active/Idle) \end{itemize} \end{itemize} \item {} \sphinxAtStartPar LINKS: \sphinxstyleemphasis{Actual} between two nodes and \sphinxstyleemphasis{Potential} between UEs and all neighbor cells \begin{itemize} \item {} \sphinxAtStartPar Link Attributes: \begin{itemize} \item {} \sphinxAtStartPar Identifiers \item {} \sphinxAtStartPar Link Type \item {} \sphinxAtStartPar Config/Bearer Parameters \item {} \sphinxAtStartPar QCI Value \end{itemize} \end{itemize} \item {} \sphinxAtStartPar SLICES: Virtualized RAN Construct \begin{itemize} \item {} \sphinxAtStartPar Slice Attributes: \begin{itemize} \item {} \sphinxAtStartPar Links \item {} \sphinxAtStartPar Bearers/Flows \item {} \sphinxAtStartPar Validity Period \item {} \sphinxAtStartPar Desired KPIs \item {} \sphinxAtStartPar MAC RRM Configuration \item {} \sphinxAtStartPar RRM Control Configuration \end{itemize} \end{itemize} \end{itemize} \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide19}.png} \caption{Example set of control applications running on top of Near\sphinxhyphen{}Real\sphinxhyphen{}Time RAN Controller.}\label{\detokenize{ran:id5}}\label{\detokenize{ran:fig-ran-controller}}\end{figure} \sphinxAtStartPar The example Control Apps in \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}} include a range of possibilities, but is not intended to be an exhaustive list. The right\sphinxhyphen{}most example, RAN Slicing, is the most ambitious in that it introduces a new capability: Virtualizing the RAN. It is also an idea that has been implemented, which we describe in more detail in the next chapter. \sphinxAtStartPar The next three (RF Configuration, Semi\sphinxhyphen{}Persistent Scheduling, Cipher Key Assignment) are examples of configuration\sphinxhyphen{}oriented applications. They provide a programmatic way to manage seldom\sphinxhyphen{}changing configuration state, thereby enabling zero\sphinxhyphen{}touch operations. Coming up with meaningful policies (perhaps driven by analytics) is likely to be an avenue for innovation in the future. \sphinxAtStartPar The left\sphinxhyphen{}most four example Control Applications are the sweet spot for SDN, with its emphasis on central control over distributed forwarding. These functions—Link Aggregation Control, Interference Management, Load Balancing, and Handover Control—are currently implemented by individual base stations with only local visibility, but they have global consequences. The SDN approach is to collect the available input data centrally, make a globally optimal decision, and then push the respective control parameters back to the base stations for execution. Realizing this value in the RAN is still a work\sphinxhyphen{}in\sphinxhyphen{}progress, but products that take this approach are emerging. Evidence using an analogous approach to optimize wide\sphinxhyphen{}area networks over many years is compelling. \sphinxAtStartPar While the above loosely categorizes the space of potential control apps as either config\sphinxhyphen{}oriented or control\sphinxhyphen{}oriented, another possible characterization is based on the current practice of controlling the mobile link at two different levels. At a fine\sphinxhyphen{}grain level, per\sphinxhyphen{}node and per\sphinxhyphen{}link control is conducted using Radio Resource Management (RRM) functions that are distributed across the individual base stations. RRM functions include scheduling, handover control, link and carrier aggregation control, bearer control, and access control. At a coarse\sphinxhyphen{}grain level, regional mobile network optimization and configuration is conducted using \sphinxstyleemphasis{Self\sphinxhyphen{}Organizing Network (SON)} functions. These functions oversee neighbor lists, manage load balancing, optimize coverage and capacity, aim for network\sphinxhyphen{}wide interference mitigation, centrally configure parameters, and so on. As a consequence of these two levels of control, it is not uncommon to see reference to \sphinxstyleemphasis{RRM Applications} and \sphinxstyleemphasis{SON Applications}, respectively, in O\sphinxhyphen{}RAN documents for SD\sphinxhyphen{}RAN. \phantomsection\label{\detokenize{ran:reading-b4}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For an example of how SDN principles have been successfully applied to a production network, we recommend \sphinxhref{https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf}{B4: Experience with a Globally\sphinxhyphen{}Deployed Software Defined WAN}. ACM SIGCOMM, August 2013. \end{sphinxadmonition} \section{4.4 Architect to Evolve} \label{\detokenize{ran:architect-to-evolve}} \sphinxAtStartPar We conclude this description of RAN internals by re\sphinxhyphen{}visiting the sequence of steps involved in disaggregation, which as the previous three sections reveal, is being pursued in multiple tiers. In doing so, we tie up several loose ends, including the new interfaces disaggregation exposes. These interfaces define the pivot points around which 5G RAN is architected to evolve. \sphinxAtStartPar In the first tier of disaggregation, 3GPP standards provide multiple options of how horizontal RAN splits can take place. Horizontal disaggregation basically splits the RAN pipeline shown in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} into independently operating components. \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (a)} illustrates horizontal disaggregation of the RAN from a single base station into three distinct components: CU, DU and RU. The O\sphinxhyphen{}RAN Alliance has selected specific disaggregation options from 3GPP and is developing open interfaces between these components. 3GPP defines the \sphinxstylestrong{N2} and \sphinxstylestrong{N3} interfaces between the RAN and the Mobile Core. \sphinxAtStartPar The second tier of disaggregation is vertical, focusing on control/user plane separation (CUPS) of the CU, and resulting in CU\sphinxhyphen{}U and CU\sphinxhyphen{}C shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (b)}. The control plane in question is the 3GPP control plane, where the CU\sphinxhyphen{}U realizes a pipeline for user traffic and the CU\sphinxhyphen{}C focuses on control message signaling between Mobile Core and the disaggregated RAN components (as well as to the UE). The O\sphinxhyphen{}RAN specified interfaces between these disaggregated components are also shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (b)}. \sphinxAtStartPar The third tier follows the SDN paradigm by carrying vertical disaggregation one step further. It does this by separating most of RAN control (RRM functions) from the disaggregated RAN components, and logically centralizing them as applications running on an SDN Controller, which corresponds to the Near\sphinxhyphen{}RT RIC shown previously in \hyperref[\detokenize{ran:fig-rrc-split}]{Figures \ref{\detokenize{ran:fig-rrc-split}}} and \hyperref[\detokenize{ran:fig-ran-controller}]{\ref{\detokenize{ran:fig-ran-controller}}}. This SDN\sphinxhyphen{}based vertical disaggregation is repeated here in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (c)}. The figure also shows the additional O\sphinxhyphen{}RAN prescribed interfaces. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide39}.png} \caption{Three tiers of RAN disaggregation: (a) horizontal, (b) vertical CUPS, and (c) vertical SDN.}\label{\detokenize{ran:id6}}\label{\detokenize{ran:fig-disagg}}\end{figure} \sphinxAtStartPar The interface names are cryptic, and knowing their details adds little to our conceptual understanding of the RAN, except perhaps to re\sphinxhyphen{}enforce how challenging it is to introduce a transformative technology like Software\sphinxhyphen{}Defined Networking into an operational environment that is striving to achieve full backward compatibility and universal interoperability. That said, we call out two notable examples. \sphinxAtStartPar The first is the \sphinxstylestrong{A1} interface that the mobile operator’s management plane—typically called the \sphinxstyleemphasis{OSS/BSS (Operations Support System / Business Support System)} in the Telco world—uses to configure the RAN. We have not discussed the Telco OSS/BSS up to this point, but it safe to assume such a component sits at the top of any Telco software stack. It is the source of all configuration settings and business logic needed to operate a network. Notice that the Management Plane shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (c)} includes a \sphinxstyleemphasis{Non\sphinxhyphen{}Real\sphinxhyphen{}Time RIC} functional block, complementing the Near\sphinxhyphen{}RT RIC that sits below the A1 interface. We return to the relevance of these two RICs in a moment. \sphinxAtStartPar The second is the \sphinxstylestrong{E2} interface that the Near\sphinxhyphen{}RT RIC uses to control the underlying RAN elements. A requirement of the E2 interface is that it be able to connect the Near\sphinxhyphen{}RT RIC to different types of RAN elements. This range is reflected in the API, which revolves around a \sphinxstyleemphasis{Service Model} abstraction. The idea is that each RAN element advertises a Service Model, which effectively defines the set of RAN Functions the element is able to support. The RIC then issues a combination of the following four operations against this Service Model. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Report:} RIC asks the element to report a function\sphinxhyphen{}specific value setting. \item {} \sphinxAtStartPar \sphinxstylestrong{Insert:} RIC instructs the element to activate a user plane function. \item {} \sphinxAtStartPar \sphinxstylestrong{Control:} RIC instructs the element to activate a control plane function. \item {} \sphinxAtStartPar \sphinxstylestrong{Policy:} RIC sets a policy parameter on one of the activated functions. \end{itemize} \sphinxAtStartPar Of course, it is the RAN element, through its published Service Model, that defines the relevant set of functions that can be activated, the variables that can be reported, and policies that can be set. \sphinxAtStartPar Taken together, the A1 and E2 interfaces complete two of the three major control loops of the RAN: the outer (non\sphinxhyphen{}real\sphinxhyphen{}time) loop has the Non\sphinxhyphen{}RT RIC as its control point and the middle (near\sphinxhyphen{}real\sphinxhyphen{}time) loop has the Near\sphinxhyphen{}RT RIC as its control point. The third (inner) control loop, which is not shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}}}, runs inside the DU: It includes the real\sphinxhyphen{}time Scheduler embedded in the MAC stage of the RAN pipeline. The two outer control loops have rough time bounds of \textgreater{}\textgreater{}1sec and \textgreater{}10ms, respectively, and as we saw in Chapter 2, the real\sphinxhyphen{}time control loop is assumed to be \textless{}1ms. \sphinxAtStartPar This raises the question of how specific functionality is distributed between the Non\sphinxhyphen{}RT RIC, Near\sphinxhyphen{}RT RIC, and DU. Starting with the second pair (i.e., the two inner loops), it is important to recognize that not all RRM functions can be centralized. After horizontal and vertical CUPS disaggregation, the RRM functions are split between CU\sphinxhyphen{}C and DU. For this reason, the SDN\sphinxhyphen{}based vertical disaggregation focuses on centralizing CU\sphinxhyphen{}C\sphinxhyphen{}side RRM functions in the Near\sphinxhyphen{}RT RIC. In addition to RRM control, this includes all the SON applications. \sphinxAtStartPar Turning to the outer two control loops, the Near RT\sphinxhyphen{}RIC opens the possibility of introducing policy\sphinxhyphen{}based RAN control, whereby interrupts (exceptions) to operator\sphinxhyphen{}defined policies would signal the need for the outer loop to become involved. For example, one can imagine developing learning\sphinxhyphen{}based controls, where the inference engines for these controls would run as applications on the Near RT\sphinxhyphen{}RIC, and their non\sphinxhyphen{}real\sphinxhyphen{}time learning counterparts would run elsewhere. The Non\sphinxhyphen{}RT RIC would then interact with the Near\sphinxhyphen{}RT RIC to deliver relevant operator policies from the Management Plane to the Near RT\sphinxhyphen{}RIC over the A1 interface. \sphinxAtStartPar Finally, you may be wondering why there is an O\sphinxhyphen{}RAN Alliance in the first place, given that 3GPP is already the standardization body responsible for interoperability across the global cellular network. The answer is that over time 3GPP has become a vendor\sphinxhyphen{}dominated organization, whereas O\sphinxhyphen{}RAN was created more recently by network operators. (AT\&T and China Mobile were the founding members.) O\sphinxhyphen{}RAN’s goal is to catalyze a software\sphinxhyphen{}based implementation that breaks the vendor lock\sphinxhyphen{}in that dominates today’s marketplace. The E2 interface in particular, which is architected around the idea of supporting different Service Models, is central to this strategy. Whether the operators will be successful in their ultimate goal is yet to be seen. \chapter{Chapter 5: Advanced Capabilities} \label{\detokenize{disaggregate:chapter-5-advanced-capabilities}}\label{\detokenize{disaggregate::doc}} \sphinxAtStartPar Disaggregating the cellular network pays dividends. This chapter explores three examples. Stepping back to look at the big picture, Chapter 3 (Architecture) described “what is” (the basics of 3GPP) and Chapter 4 (RAN Internals) described “what will be” (where the industry is clearly headed), whereas this chapter describes “what might be” (our best judgement on cutting\sphinxhyphen{}edge capabilities that will eventually be realized). \section{5.1 Optimized Data Plane} \label{\detokenize{disaggregate:optimized-data-plane}} \sphinxAtStartPar There are many reasons to disaggregate functionality, but one of the most compelling is that decoupling control and data code paths allows them to be optimized independently. The data path, for example, can be optimized by programming it into specialized hardware. Modern white\sphinxhyphen{}box switches with programmable packet forwarding pipelines are a perfect example of specialized hardware we can exploit in the cellular network. \hyperref[\detokenize{disaggregate:fig-e2e}]{Figure \ref{\detokenize{disaggregate:fig-e2e}}} shows the first step in the process of doing this. The figure also pulls together all the elements we’ve been describing up to this point. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide21}.png} \caption{End\sphinxhyphen{}to\sphinxhyphen{}end disaggregated system, including Mobile Core and Split\sphinxhyphen{}RAN.}\label{\detokenize{disaggregate:id1}}\label{\detokenize{disaggregate:fig-e2e}}\end{figure} \sphinxAtStartPar There are several things to note about this diagram. First, the figure combines both the Mobile Core and RAN elements, organized according to the major subsystems: Mobile Core, Central Unit (CU), Distributed Unit (DU), and Radio Unit (RU). The figure also shows one possible mapping of these subsystems onto physical locations, with the first two co\sphinxhyphen{}located in a Central Office and the latter two co\sphinxhyphen{}located in a cell tower. As discussed earlier, other configurations are also possible. \sphinxAtStartPar Second, the figure shows the Mobile Core’s two user plane elements (PGW, SGW) and the Central Unit’s single user plane element (PDCP) further disaggregated into control/user plane pairs, denoted PGW\sphinxhyphen{}C / PGW\sphinxhyphen{}U, SGW\sphinxhyphen{}C / SGW\sphinxhyphen{}U, and PDCP\sphinxhyphen{}C / PDCP\sphinxhyphen{}U, respectively. Exactly how this decoupling is realized is an implementation choice (i.e., not specified by 3GPP), but the idea is to reduce the User Plane component to the minimal Receive\sphinxhyphen{}Packet / Process\sphinxhyphen{}Packet / Send\sphinxhyphen{}Packet processing core, and elevate all control\sphinxhyphen{}related aspects into the Control Plane component. \sphinxAtStartPar Third, the PHY (Physical) element of the RAN pipeline is split between the DU and RU partition. Although beyond the scope of this book, the 3GPP spec specifies the PHY element as a collection of functional blocks, some of which can be effectively implemented by software running on a general\sphinxhyphen{}purpose processor and some of which are best implemented in specialized hardware (e.g., a Digital Signal Processor). These two subsets of functional blocks map to the PHY Upper (part of the DU) and the PHY Lower (part of the RU), respectively. \sphinxAtStartPar Fourth, and somewhat confusingly, \hyperref[\detokenize{disaggregate:fig-e2e}]{Figure \ref{\detokenize{disaggregate:fig-e2e}}} shows the PDCP\sphinxhyphen{}C element and the Control Plane (Forwarding) element combined into a single functional block, with a data path (blue line) connecting that block to both the RLC and the MME. Exactly how this pair is realized is an implementation choice (e.g., they could map onto two or more microservices), but the end result is that they are part of an end\sphinxhyphen{}to\sphinxhyphen{}end path over which the MME can send control packets to the UE. Note that this means responsibility for demultiplexing incoming packets between the control plane and user plane falls to the RLC. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide22}.png} \caption{Implementing data plane elements of the User Plane in programmable switches.}\label{\detokenize{disaggregate:id2}}\label{\detokenize{disaggregate:fig-e2e-p4}}\end{figure} \sphinxAtStartPar \hyperref[\detokenize{disaggregate:fig-e2e-p4}]{Figure \ref{\detokenize{disaggregate:fig-e2e-p4}}} shows why we disaggregated these components: it allows us to realize the three user plane elements (PGW\sphinxhyphen{}U, SGW\sphinxhyphen{}U, PDCP\sphinxhyphen{}U) in switching hardware. This can be done using a combination of a language that is tailored for programming forwarding pipelines (e.g., P4), and a protocol\sphinxhyphen{}independent switching architecture (e.g., Intel’s Tofino). For now, the important takeaway is that the RAN and Mobile Core user plane can be mapped directly onto an SDN\sphinxhyphen{}enabled data plane. \phantomsection\label{\detokenize{disaggregate:reading-p4}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For more information about P4 and programmable switching chips, we recommend \sphinxhref{https://sdn.systemsapproach.org/switch.html}{White\sphinxhyphen{}Box Switches}, a chapter in \sphinxstyleemphasis{Software\sphinxhyphen{}Defined Networking: A Systems Approach}, March 2020. \end{sphinxadmonition} \sphinxAtStartPar Pushing RAN and Mobile Core forwarding functionality into the switching hardware results in overlapping terminology that can be confusing. 5G separates the functional blocks into control and user planes, while SDN disaggregates a given functional block into control and data plane halves. The overlap comes from our choosing to implement the 5G user plane by splitting it into its SDN\sphinxhyphen{}based control and data plane parts. As one simplification, we refer to the Control Plane (Forwarding) and PDCP\sphinxhyphen{}C combination as the CU\sphinxhyphen{}C (Central Unit \sphinxhyphen{} Control) going forward. \sphinxAtStartPar Finally, the SDN\sphinxhyphen{}prescribed control/data plane disaggregation comes with an implied implementation strategy, namely, the use of a scalable and highly available \sphinxstyleemphasis{Network Operating System (NOS)}. Like a traditional OS, a NOS sits between application programs (control apps) and the underlying hardware devices (whitebox switches), providing higher levels abstractions (e.g., network graph) to those applications, while hiding the low\sphinxhyphen{}level details of the underlying hardware. To make the discussion more concrete, we use ONOS (Open Network Operating System) as an example NOS, where PGW\sphinxhyphen{}C, SGW\sphinxhyphen{}C, and PDCP\sphinxhyphen{}C are all realized as control applications running on top of ONOS. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide23}.png} \caption{Control Plane elements of the User Plane implemented as Control Applications running on an SDN Controller (e.g., ONOS).}\label{\detokenize{disaggregate:id3}}\label{\detokenize{disaggregate:fig-onos}}\end{figure} \sphinxAtStartPar \hyperref[\detokenize{disaggregate:fig-onos}]{Figure \ref{\detokenize{disaggregate:fig-onos}}} shows one possible configuration in which the underlying switches are interconnected to form a leaf\sphinxhyphen{}spine fabric. Keep in mind that the linear sequence of switches implied by \hyperref[\detokenize{disaggregate:fig-e2e-p4}]{Figure \ref{\detokenize{disaggregate:fig-e2e-p4}}} is logical, but that in no way restricts the actual hardware to the same topology. The reason we use a leaf\sphinxhyphen{}spine topology is related to our ultimate goal of building an edge cloud, and leaf\sphinxhyphen{}spine is the prototypical structure for such cloud\sphinxhyphen{}based clusters. This means the three control applications must work in concert to implement an end\sphinxhyphen{}to\sphinxhyphen{}end path through the fabric, which in practice happens with the aid of other, fabric\sphinxhyphen{}aware, control applications (as implied by the “…” in the Figure). We describe the complete picture in more detail in the next chapter, but for now, the big takeaway is that the control plane components of the 5G overlay can be realized as control applications for an SDN\sphinxhyphen{}based underlay. \section{5.2 Multi\sphinxhyphen{}Cloud} \label{\detokenize{disaggregate:multi-cloud}} \sphinxAtStartPar Another consequence of disaggregating functionality is that once decoupled, different functions can be placed in different physical locations. We have already seen this when we split the RAN, placing some functions (e.g., the PDCP and RRC) in the Central Unit and others (e.g., RLC and MAC) in Distributed Units. This allows for simpler (less expensive) hardware in remote locations, where there are often space, power, and cooling constraints. \sphinxAtStartPar This process can be repeated by distributing the more centralized elements across multiple clouds, including large datacenters that already benefit from elasticity and economies of scale. \hyperref[\detokenize{disaggregate:fig-multicloud}]{Figure \ref{\detokenize{disaggregate:fig-multicloud}}} shows the resulting multi\sphinxhyphen{}cloud realization of the Mobile Core. We leave the user plane at the edge of the network (e.g., in the Central Office) and move control plane to a centralized cloud. It could even be a public cloud like Google or Amazon. This includes not only the MME, PCRF, and HSS, but also the PGW\sphinxhyphen{}C and SGW\sphinxhyphen{}C we decoupled in the previous section. (Note that \hyperref[\detokenize{disaggregate:fig-multicloud}]{Figure \ref{\detokenize{disaggregate:fig-multicloud}}} renames the PDCP\sphinxhyphen{}U from earlier diagrams as the CU\sphinxhyphen{}U; either label is valid.) \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide24}.png} \caption{Multi\sphinxhyphen{}Cloud implementation, with MME, HSS, PCRF and Control Plane elements of the PGW and SGW running in a centralized cloud.}\label{\detokenize{disaggregate:id4}}\label{\detokenize{disaggregate:fig-multicloud}}\end{figure} \sphinxAtStartPar What is the value in doing this? Just like the DU and RU, the Edge Cloud likely has limited resources. If we want room to run new edge services there, it helps to move any components that need not be local to a larger facility with more abundant resources. Centralization also facilitates collecting and analyzing data across multiple edge locations, which is harder to do if that information is distributed over multiple sites. (Analytics performed on this data also benefits from having abundant compute resources available.) \sphinxAtStartPar Another reason worth calling out is that it lowers the barrier for anyone (not just the companies that own and operate the RAN infrastructure) to offer mobile services to customers. These entities are called \sphinxstyleemphasis{MVNOs (Mobile Virtual Network Operators)} and one clean way to engineer an MVNO is to run your own Mobile Core in a cloud of your choosing. \sphinxAtStartPar But the biggest motivation for the configuration shown in \hyperref[\detokenize{disaggregate:fig-multicloud}]{Figure \ref{\detokenize{disaggregate:fig-multicloud}}} is that keeping the user plane elements of the mobile core at the edge makes it possible to “break out” local traffic without having to traverse a “hairpin” route that goes through a central site. This has the potential to dramatically reduce latency for any edge\sphinxhyphen{}hosted services. We return to this topic in Chapter 7. \section{5.3 Network Slicing} \label{\detokenize{disaggregate:network-slicing}} \sphinxAtStartPar One of the most compelling value propositions of 5G is the ability to differentiate the level of service offered to different applications and customers. Differentiation, of course, is key to being able to charge some customers more than others, but the monetization case is only part of the story. It is also necessary to support applications with widely varying requirements. For example, streaming video requires high bandwidth but can tolerate larger latencies, while IoT has minimal bandwidth needs but sometimes requires extremely low and predictable latencies, and entails connecting a \sphinxstyleemphasis{massively scalable} number of devices. \sphinxAtStartPar The mechanism that supports this sort of differentiation is called network slicing, and it fundamentally comes down to scheduling, both in the RAN (deciding which segments to transmit) and in the Mobile Core (scaling microservice instances and placing those instances on the available servers). The following introduces the basic idea, starting with the RAN. \sphinxAtStartPar But before getting into the details, we note that a network slice is a realization of the QoS Class Identifier (QCI) discussed earlier. 3GPP specifies a standard set of network slices, called \sphinxstyleemphasis{Standardized Slice Type (SST)} values. For example, SST 1 corresponds to mobile broadband, SST 2 corresponds to Ultra\sphinxhyphen{}Reliable Low Latency Communications, SST 3 corresponds to Massive IoT, and so on. It is also possible to extend this standard set with additional slice behaviors, as well as define multiple slices for each SST (e.g., to further differentiate subscribers based on priority). \sphinxAtStartPar Of course, defining a desired set of slices and implementing a slicing mechanism are two entirely different things. The following sketches how slices can be implemented. \subsection{RAN Slicing} \label{\detokenize{disaggregate:ran-slicing}} \sphinxAtStartPar We start by reviewing the basic scheduling challenge previewed in Chapter 2. As depicted in \hyperref[\detokenize{disaggregate:fig-slice-sched}]{Figure \ref{\detokenize{disaggregate:fig-slice-sched}}}, the radio spectrum can be conceptualized as a 2\sphinxhyphen{}D grid of \sphinxstyleemphasis{Resource Blocks (RB)}, where the scheduler’s job is to decide how to fill the grid with the available segments from each user’s transmission queue based on CQI feedback from the UEs. To restate, the power of OFDMA is the flexibility it provides in how this mapping is performed. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide27}.png} \caption{Scheduler allocating resource blocks to UEs.}\label{\detokenize{disaggregate:id5}}\label{\detokenize{disaggregate:fig-slice-sched}}\end{figure} \sphinxAtStartPar While in principle one could define an uber scheduler that takes dozens of different factors into account, the key to network slicing is to add a layer of indirection. As shown in \hyperref[\detokenize{disaggregate:fig-hypervisor}]{Figure \ref{\detokenize{disaggregate:fig-hypervisor}}}, the idea is to perform a second mapping of Virtual RBs to Physical RBs. This sort of virtualization is common in resource allocators throughout computing systems because we want to separate how many resources are allocated to each user from the decision as to which physical resources are actually assigned. This virtual\sphinxhyphen{}to\sphinxhyphen{}physical mapping is performed by a layer typically known as a \sphinxstyleemphasis{Hypervisor}, and the important thing to keep in mind is that it is totally agnostic about which user’s segment is affected by each translation. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide28}.png} \caption{Wireless Hypervisor mapping virtual resource blocks to physical resource blocks.}\label{\detokenize{disaggregate:id6}}\label{\detokenize{disaggregate:fig-hypervisor}}\end{figure} \sphinxAtStartPar Having decoupled the Virtual RBs from Physical RBs, it is now possible to define multiple Virtual RB sets (of varying sizes), each with its own scheduler. \hyperref[\detokenize{disaggregate:fig-multi-sched}]{Figure \ref{\detokenize{disaggregate:fig-multi-sched}}} gives an example with two equal\sphinxhyphen{}sized RB sets, where the important consequence is that having made the macro\sphinxhyphen{}decision that the Physical RBs are divided into two equal partitions, the scheduler associated with each partition is free to allocate Virtual RBs completely independent from each other. For example, one scheduler might be designed to deal with high\sphinxhyphen{}bandwidth video traffic and another scheduler might be optimized for low\sphinxhyphen{}latency IoT traffic. Alternatively, a certain fraction of the available capacity could be reserved for premium customers or other high\sphinxhyphen{}priority traffic (e.g., public safety), with the rest shared among everyone else. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide29}.png} \caption{Multiple schedulers running on top of wireless hypervisor.}\label{\detokenize{disaggregate:id7}}\label{\detokenize{disaggregate:fig-multi-sched}}\end{figure} \sphinxAtStartPar Going one level deeper in the implementation details, the real\sphinxhyphen{}time scheduler running in each DU receives high\sphinxhyphen{}level directives from the near\sphinxhyphen{}real\sphinxhyphen{}time scheduler running in the CU, and as depicted in \hyperref[\detokenize{disaggregate:fig-slicing-control}]{Figure \ref{\detokenize{disaggregate:fig-slicing-control}}}, the DUs follow these directives in making their scheduling decisions \sphinxstyleemphasis{on a per\sphinxhyphen{}slice basis}. A single RAN Slicing control application is responsible for the macro\sphinxhyphen{}scheduling decision by allocating resources among a set of slices. Understanding this implementation detail is important because all of these control decisions are implemented by software modules, and hence, easily changed or customized. They are not “locked” into the underlying system, as they have historically been in 4G’s eNodeBs. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=350\sphinxpxdimen]{{Slide30}.png} \caption{Centralized near\sphinxhyphen{}real\sphinxhyphen{}time control applications cooperating with distributed real\sphinxhyphen{}time RAN schedulers.}\label{\detokenize{disaggregate:id8}}\label{\detokenize{disaggregate:fig-slicing-control}}\end{figure} \sphinxAtStartPar In summary, the goal of RAN slicing is to programmatically create virtual RAN nodes (base stations) that operate on the same hardware and share the spectrum resources according to a given policy for different applications, services, users, and so on. Tying RAN slicing back to RAN disaggregation, one can imagine several possible configurations, depending on the desired level of isolation between the slices. \hyperref[\detokenize{disaggregate:fig-ran-slicing}]{Figure \ref{\detokenize{disaggregate:fig-ran-slicing}}} shows four examples, all of which assume slices share the antennas and RF components, which is effectively the RU: (a) RAN slices share RU, DU, CU\sphinxhyphen{}U and CU\sphinxhyphen{}C; (b) RAN slices share RU and DU, but have their own CU\sphinxhyphen{}U and CU\sphinxhyphen{}C; (c) RAN slices share RU, CU\sphinxhyphen{}U and CU\sphinxhyphen{}C, but have their own DU; and (d) RAN slices share RU, but have their own DU, CU\sphinxhyphen{}U and CU\sphinxhyphen{}C. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide40}.png} \caption{Four possible configurations of a disaggregated RAN in support of RAN slicing.}\label{\detokenize{disaggregate:id9}}\label{\detokenize{disaggregate:fig-ran-slicing}}\end{figure} \subsection{Core Slicing} \label{\detokenize{disaggregate:core-slicing}} \sphinxAtStartPar In addition to slicing the RAN, we also need to slice the Mobile Core. In many ways, this is a well\sphinxhyphen{}understood problem, involving QoS mechanisms in the network switches (i.e., making sure packets flow through the switching fabric according to the bandwidth allocated to each slice) and the cluster processors (i.e., making sure the containers that implement each microservice are allocated sufficient CPU cores to sustain the packet forwarding rate of the corresponding slice). \sphinxAtStartPar But packet scheduling and CPU scheduling are low\sphinxhyphen{}level mechanisms. What makes slicing work is to also virtualize and replicate the entire service graph that implements the Mobile Core. If you think of a slice as a system abstraction, then that abstraction needs to keep track of the set of interconnected microservices that implement each slice, and then instruct the underlying packet schedulers to allocate sufficient network bandwidth to the slice’s flows and the underlying CPU schedulers to allocate sufficient compute cycles to the slice’s containers. \sphinxAtStartPar For example, if there are two network slices (analogous to the two RAN schedulers shown in \hyperref[\detokenize{disaggregate:fig-multi-sched}]{Figures \ref{\detokenize{disaggregate:fig-multi-sched}}} and \hyperref[\detokenize{disaggregate:fig-slicing-control}]{\ref{\detokenize{disaggregate:fig-slicing-control}}}), then there would also need to be two Mobile Core service graphs: One set of AMF, SMF, UPF,… microservices running on behalf of the first slice and a second set of AMF, SMF, UPF,… microservices running on behalf of the second slice. These two graphs would scale independently (i.e., include a different number of container instances), depending on their respective workloads and QoS guarantees. The two slices would also be free to make different implementation choices, for example, with one optimized for massive IoT applications and the other optimized for high\sphinxhyphen{}bandwidth AR/VR applications. \sphinxAtStartPar The one remaining mechanism we need is a demultiplexing function that maps a given packet flow (e.g., between UE and some Internet application) onto the appropriate instance of the service graph. This is the job of the NSSF described in an Chapter 3: it is responsible for selecting the instance a given slice’s traffic is to traverse. \sphinxAtStartPar The similarity between slicing and the much\sphinxhyphen{}debated topic of network QoS might lead one to conclude that slicing won’t take off, as QoS never seemed to provide quite enough benefit in large networks to justify its complexity. But the one place where QoS \sphinxstyleemphasis{is} effective is when bandwidth is scarce, e.g., in last\sphinxhyphen{}mile links. For this reason, slicing is likely to be valuable precisely because it allows efficient partitioning of the relatively scarce resource that is cellular spectrum. \sphinxAtStartPar Finally, note that slicing can be viewed as a way to enable overlays on the cellular network. The Internet has a long history of supporting innovation through the use of overlays. Many innovations such as multicast and content delivery networks were developed in this way. So while we might not know exactly how slicing will be used, it is an important ingredient in fostering innovation. \chapter{Chapter 6: Exemplar Implementation} \label{\detokenize{impl:chapter-6-exemplar-implementation}}\label{\detokenize{impl::doc}} \sphinxAtStartPar The steps we’ve taken in the previous chapters to virtualize, disaggregate, optimize, distribute, and slice the cellular network not only help us understand the inner\sphinxhyphen{}workings of 5G, but they are also necessary to reduce the entirety of the 5G cellular network to practice. The goal is an implementation, which by definition, forces us to make specific engineering choices. This chapter describes one set of engineering choices that results in a running system. It should be interpreted as an exemplar, for the sake of completeness, but not the only possibility. \sphinxAtStartPar The system we describe is called CORD, which you will recall from the Introduction is an acronym (\sphinxstylestrong{C}entral \sphinxstylestrong{O}ffice \sphinxstylestrong{R}e\sphinxhyphen{}architected as a \sphinxstylestrong{D}atacenter). More concretely, CORD is a blueprint for building a 5G deployment from commodity hardware and a collection of open source software components. We call this hardware/software combination a CORD POD, where the idea is to deploy a POD at each edge site that is part of a cellular network. The following describes CORD in terms of a set of engineering decisions. It is not a substitute for detailed documentation for installing, developing, and operating CORD. Also keep in mind that even though CORD includes “Central Office” in its name, a CORD POD is a general design, and not strictly limited to being deployed in a conventional Central Office. \phantomsection\label{\detokenize{impl:reading-guide}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar To learn how to install, operate, and contribute to the CORD open source software platform, see the \sphinxhref{https://guide.opencord.org}{CORD Guide}. ONF, March 2020. \end{sphinxadmonition} \sphinxAtStartPar Before getting into the specifics, it is important to understand that CORD is a work\sphinxhyphen{}in\sphinxhyphen{}progress, with a sizable open source community contributing to its code base. Many of the components are quite mature, and currently running in operator trials and production networks. Others (largely corresponding to the advanced capabilities described in the previous chapter) are prototypes that run in “demonstration mode,” but are not yet complete enough to be included in official releases. Also, as outlined in the earlier discussion on deployment options, CORD starts with a production\sphinxhyphen{}quality EPC that is being incrementally evolved into its 5G counterpart. (This chapter uses the EPC\sphinxhyphen{}specific components for illustrative purposes.) \section{6.1 Framework} \label{\detokenize{impl:framework}} \sphinxAtStartPar \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} gives a schematic overview of a CORD POD. It connects downstream to a set of DUs (and associated RUs), and upstream to the rest of the Internet. Internally, it includes a set of commodity servers (the figure shows four racks of three servers each, but the design accommodates anywhere from a partial rack to 16 racks) and a set of white\sphinxhyphen{}box switches arranged in a leaf\sphinxhyphen{}spine topology (the figure shows two leaves and two spine switches, but the design allows anywhere from a single switch to two leaf switches per rack and as many spine switches as necessary to provide sufficient east\sphinxhyphen{}to\sphinxhyphen{}west bandwidth). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=650\sphinxpxdimen]{{Slide25}.png} \caption{CORD implementation of RAN and Mobile Core.}\label{\detokenize{impl:id1}}\label{\detokenize{impl:fig-cord}}\end{figure} \sphinxAtStartPar With respect to software, \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} shows a combination of RAN (red) and Mobile Core (blue) components, plus the modules that define the CORD platform (orange). We describe the platform components later in this chapter, but you can think of them as collectively implementing a multi\sphinxhyphen{}tenant cloud on top of which many different scalable services can run. The RAN and Mobile Core are two such tenants. The CORD platform can also host other edge services (which is one reason CORD is built using cloud technology in the first place), but exactly what other edge services might run on a given CORD POD is a question we do not try to answer in this book. \sphinxAtStartPar The RAN and Core related components are ones we’ve described in earlier chapters. They include the Control and User planes of the CU and Mobile Core, respectively. To simplify the diagram, we show the SGW and PGW merged into a single S/PGW. One other detail that deserves closer attention is the RAN Control component included in the CU Control Plane. This is the Near\sphinxhyphen{}RT RIC introduced in Section 4.3, which means a CORD POD includes two SDN Controllers: the RIC controls the RAN and ONOS (shown in \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}}) controls the fabric. (The RIC running in CORD actually happens to be a second, customized version of ONOS, but that’s an implementation detail.) \sphinxAtStartPar One aspect of \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} that requires further elaboration is how each of the RAN and Mobile Core components are actually realized. Specifically, there are three different manifestations of the functional components implied by the figure: (1) the data plane layer of the CU\sphinxhyphen{}U and S/PGW\sphinxhyphen{}U are realized as P4 programs loaded into the programmable switches; (2) the control plane layer of the CU\sphinxhyphen{}U and S/PGW\sphinxhyphen{}U (as well as the Trellis platform module) are realized as control applications loaded onto ONOS; and (3) the remaining components are realized as Kubernetes\sphinxhyphen{}managed microservices. (Kubernetes is implied, but not shown in the figure.) \sphinxAtStartPar To expand on this idea, \hyperref[\detokenize{impl:fig-ci-cd}]{Figure \ref{\detokenize{impl:fig-ci-cd}}} gives an alternative view of a CORD POD, abstracting away the details of \sphinxstyleemphasis{what} services it hosts, and focusing instead on \sphinxstyleemphasis{how} those services are instantiated on the POD. In this figure, all the functionality instantiated onto the POD runs as a combination of Kubernetes\sphinxhyphen{}based microservices and ONOS\sphinxhyphen{}based control applications. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=350\sphinxpxdimen]{{Slide26}.png} \caption{Alternative view of CORD, with a CI/CD toolchain managing the platform and set of services implemented by a combination of ONOS\sphinxhyphen{}based control apps and Kubernetes\sphinxhyphen{}based microservices.}\label{\detokenize{impl:id2}}\label{\detokenize{impl:fig-ci-cd}}\end{figure} \sphinxAtStartPar When abstracted in this way, we can view a POD as including three major subsystems. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Platform:} The base layer common to all configurations includes Kubernetes as the container management system and ONOS as the SDN controller, with Stratum (an open source switch OS) loaded on to each switch. Both ONOS and the control applications it hosts run as Kubernetes\sphinxhyphen{}managed microservices. \item {} \sphinxAtStartPar \sphinxstylestrong{Profile:} The deployment\sphinxhyphen{}specific collection of microservices and SDN control apps that have been selected to run on a particular POD. This is a variable and evolvable set, and it includes the control plane and edge services described elsewhere. \item {} \sphinxAtStartPar \sphinxstylestrong{CI/CD Toolchain:} Used to assemble, deploy, operate, and upgrade a particular Platform/Profile combination. It implements a set of processes that transforms a collection of disaggregated and virtualized components into an operational system capable of responding to operator directives and carrying live traffic. \end{itemize} \sphinxAtStartPar Although beyond the scope of this book, the CI/CD toolchain uses standard DevOps tools to bootstrap software onto the cluster of servers and switches, and rollout/rollback individual microservices and control applications. It also auto\sphinxhyphen{}generates the Northbound Interface (NBI) that operators use to manage the POD, based on a declarative specification of the Profile the POD is configured to support. This NBI is sufficiently complete to operate a CORD POD in a production environment. \section{6.2 Platform Components} \label{\detokenize{impl:platform-components}} \sphinxAtStartPar We now return to the three platform\sphinxhyphen{}related components shown in \hyperref[\detokenize{impl:fig-cord}]{Figures \ref{\detokenize{impl:fig-cord}}} and \hyperref[\detokenize{impl:fig-ci-cd}]{\ref{\detokenize{impl:fig-ci-cd}}}. Each is a substantial open source component in its own right, but for our purposes, it is enough to understand the role they play in supporting a 5G\sphinxhyphen{}based profile of CORD. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Stratum:} A thin operating system that runs locally on each white\sphinxhyphen{}box switch. Its purpose is to provide a hardware\sphinxhyphen{}independent interface for managing and programming the switches in CORD. This includes using \sphinxstyleemphasis{P4} to define the forwarding behavior of the switch’s forwarding pipeline (think of this program as a contract between the control and data planes), and \sphinxstyleemphasis{P4Runtime} to control that forwarding contract at runtime. \item {} \sphinxAtStartPar \sphinxstylestrong{ONOS:} A Network Operating System used to configure and control a network of programmable white\sphinxhyphen{}box switches. It runs off\sphinxhyphen{}switch as a logically centralized SDN controller, and hosts a collection of SDN control applications, each of which controls some aspect of the underlying network. Because it is logically centralized, ONOS is designed to be highly available and to have scalable performance. \item {} \sphinxAtStartPar \sphinxstylestrong{Trellis:} An ONOS\sphinxhyphen{}hosted SDN control application that implements a leaf\sphinxhyphen{}spine fabric on a network of white\sphinxhyphen{}box switches. It implements the control plane for several features, including VLANs and L2 bridging, IPv4 and IPv6 unicast and multicast routing, DHCP L3 relay, dual\sphinxhyphen{}homing of servers and upstream routers, QinQ forwarding/termination, MPLS pseudowires, and so on. In addition, Trellis can make the entire fabric appear as a single (virtual) router to upstream routers, which communicate with Trellis using standard BGP. \end{itemize} \sphinxAtStartPar Stratum (running on each switch) and ONOS (running off\sphinxhyphen{}switch and managing a network of switches) communicate using the following interfaces. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{P4:} Defines the forwarding behavior for programmable switching chips as well as modeling fixed\sphinxhyphen{}function ASIC pipelines. A P4 program defines a contract that is implemented by the data plane and programmable by the control plane. \item {} \sphinxAtStartPar \sphinxstylestrong{P4Runtime:} An SDN\sphinxhyphen{}ready interface for controlling forwarding behavior at runtime. It is the key for populating forwarding tables and manipulating forwarding state, and it does so in way that is hardware\sphinxhyphen{}agnostic. \item {} \sphinxAtStartPar \sphinxstylestrong{OpenConfig Models:} Define a base for device configuration and management. These models can be programmatically extended for platform\sphinxhyphen{}specific functionality, but the goal is to minimize model deviations so as to enable a vendor\sphinxhyphen{}agnostic management plane. \item {} \sphinxAtStartPar \sphinxstylestrong{gNMI} (gRPC Network Management Interface): Improves on the existing configuration interfaces by using a binary representation on the wire and enabling bi\sphinxhyphen{}directional streaming. Paired with the OpenConfig models, gNMI is SDN\sphinxhyphen{}ready. \item {} \sphinxAtStartPar \sphinxstylestrong{gNOI} (gRPC Network Operations Interfaces): A collection of microservices that enable switch specific operations, like certificate management, device testing, software upgrade, and networking troubleshooting. gNOI provides a semantically rich API that replaces existing CLI\sphinxhyphen{}based approaches. \end{itemize} \sphinxAtStartPar Trellis, as an SDN control application running on top of ONOS, controls packet forwarding across the switching fabric internal to a CORD POD (i.e., within a single site). But Trellis can also be extended across multiple sites deeper into the network using multiple stages of spines, as shown in \hyperref[\detokenize{impl:fig-trellis}]{Figure \ref{\detokenize{impl:fig-trellis}}}. This means Trellis has the potential to play a role in implementing the backhaul and midhaul network for the RAN, or alternatively, extending the RAN into customer premises (denoted “On Site” in the figure). \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide31}.png} \caption{Trellis control application managing a (possibly distributed) leaf\sphinxhyphen{}spine fabric.}\label{\detokenize{impl:id3}}\label{\detokenize{impl:fig-trellis}}\end{figure} \sphinxAtStartPar The software stack we’ve just described is substantial, and has the potential to disrupt the way cellular networks are built and operated. Of particular note, the RAN Intelligent Controller shown in \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} is implemented as a set of extensions to ONOS. This puts the ONOS\sphinxhyphen{}based RIC at the very center of the design, where the SDN and 5G worlds intersect. \sphinxAtStartPar This discussion, while focused on just one option for implementing 5G networks, illustrates one of the reasons 5G is viewed as such a transformation of the Telco industry. The 5G architecture, far more than any previous Telco network, capitalizes on several significant, broad industry trends. These include the rise of SDN, the power of open source software and its increasing use in networking, and of course the adoption of cloud technologies as the basis for delivering innovative services. \phantomsection\label{\detokenize{impl:reading-sdn}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For more information about the SDN software stack, we recommend a companion book: \sphinxhref{https://sdn.systemsapproach.org/}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. March 2020. \end{sphinxadmonition} \chapter{Chapter 7: Cloudification of Access} \label{\detokenize{cloud:chapter-7-cloudification-of-access}}\label{\detokenize{cloud::doc}} \sphinxAtStartPar The previous chapters went step\sphinxhyphen{}by\sphinxhyphen{}step, first breaking 5G down into its elemental components and then showing how those components could be put back together using best practices in cloud design to build a fully functional, 3GPP\sphinxhyphen{}compliant 5G cellular network. In doing so, it is easy to lose sight of the big picture, which is that the cellular network is undergoing a dramatic transformation. That’s the whole point of 5G. We conclude by making some observations about this big picture. \section{7.1 Multi\sphinxhyphen{}Cloud} \label{\detokenize{cloud:multi-cloud}} \sphinxAtStartPar To understand the impact of cloud technologies and practices being applied to the access network, it is helpful to first understand what’s important about the cloud. The cloud has fundamentally changed the way we compute, and more importantly, the pace of innovation. It has done this through a combination of the following. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Disaggregation:} Breaking vertically integrated systems into independent components with open interfaces. \item {} \sphinxAtStartPar \sphinxstylestrong{Virtualization:} Being able to run multiple independent copies of those components on a common hardware platform. \item {} \sphinxAtStartPar \sphinxstylestrong{Commoditization:} Being able to elastically scale those virtual components across commodity hardware bricks as workload dictates. \end{itemize} \sphinxAtStartPar There is an opportunity for the same to happen with the access network, or from another perspective, for the cloud to essentially expand so far as to subsume the access network. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide32}.png} \caption{A collection of multi\sphinxhyphen{}tenant clouds—including virtualized RAN resources alongside conventional compute, storage, and network resources—hosting both Telco and Over\sphinxhyphen{}the\sphinxhyphen{}Top (OTT) services and applications.}\label{\detokenize{cloud:id1}}\label{\detokenize{cloud:fig-cloud}}\end{figure} \sphinxAtStartPar \hyperref[\detokenize{cloud:fig-cloud}]{Figure \ref{\detokenize{cloud:fig-cloud}}} gives a high\sphinxhyphen{}level overview of how the transformation might play out, with the global cloud spanning edge clouds, private Telco clouds, and the familiar public clouds. We call this collection of clouds “multi\sphinxhyphen{}cloud” (although note that there are a number of other definitions for that term). Each individual cloud site is potentially owned by a different organization (this includes the cell towers, as well), and as a consequence, each site will likely be multi\sphinxhyphen{}tenant in that it is able to host (and isolate) applications on behalf of many other people and organizations. Those applications, in turn, will include a combination of the RAN and Core services (as described throughout this book), Over\sphinxhyphen{}the\sphinxhyphen{}Top (OTT) applications commonly found today in public clouds (but now also distributed across edge clouds), and new Telco\sphinxhyphen{}managed applications (also distributed across centralized and edge locations). \sphinxAtStartPar Eventually, we can expect common APIs to emerge, lowering the barrier for anyone (not just today’s network operators or cloud providers) to deploy applications across multiple sites by acquiring the storage, compute, networking, and connectivity resources they need. \section{7.2 EdgeCloud\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service} \label{\detokenize{cloud:edgecloud-as-a-service}} \sphinxAtStartPar Of all the potential outcomes discussed in the previous section, one that is rapidly gaining traction is to run a 5G\sphinxhyphen{}enabled edge cloud as a centrally managed service. As illustrated in \hyperref[\detokenize{cloud:fig-edgecloud}]{Figure \ref{\detokenize{cloud:fig-edgecloud}}}, the idea is to deploy an edge cloud in enterprises, configured with the user plane components of the RAN and Mobile Core (along with any edge services the enterprise wants to run locally), and then manage that edge deployment from the central cloud. The central cloud would run a management portal for the edge cloud, along with the control plane of the Mobile Core. This is similar to the multi\sphinxhyphen{}cloud configuration discussed in Section 5.2, except with the added feature of being able to manage multiple edge deployments from one central location. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide36}.png} \caption{EdgeCloud\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service, a managed service, with RAN and Mobile Core user plane components running in the enterprise, and the control plane of the Mobile Core (along with a management portal) running centrally in the public cloud.}\label{\detokenize{cloud:id2}}\label{\detokenize{cloud:fig-edgecloud}}\end{figure} \sphinxAtStartPar The value of such a deployment is to bring 5G wireless advantages into the enterprise, including support for predictable, low\sphinxhyphen{}latency communication required for real\sphinxhyphen{}time controlling of large numbers of mobile devices. Factory automation is one compelling use case for such an edge cloud, but interest in supporting IoT in general is giving ECaaS significant momentum. \sphinxAtStartPar This momentum has, not surprisingly, led to recent commercial activity. But there is also an open source variant, called Aether, now available for early adopters to evaluate and experiment with. Aether is an ONF\sphinxhyphen{}operated ECaaS with 4G/5G support, built from the open source components described throughout this book. Aether works with both licensed and unlicensed frequency bands (e.g., CBRS), but it is the latter that makes it an easy system to opt into. \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}} depicts the early stages of Aether’s centrally managed, multi\sphinxhyphen{}site deployment. \begin{figure}[ht] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide37}.png} \caption{Aether is an ONF\sphinxhyphen{}operated EdgeCloud\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service built from the SD\sphinxhyphen{}RAN and disaggregated Mobile Core components described throughout this book. Aether includes a centralized operations portal running in the Public Cloud.}\label{\detokenize{cloud:id3}}\label{\detokenize{cloud:fig-aether}}\end{figure} \sphinxAtStartPar Note that each edge site in \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}} corresponds to a CORD POD described in Chapter 6, re\sphinxhyphen{}configured to offload the O\&M Interface and the Control elements of the Mobile Core to the central cloud. \phantomsection\label{\detokenize{cloud:reading-aether}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar For more information about Aether, visit the \sphinxhref{https://www.opennetworking.org/aether/}{Aether Web Site}. ONF, March 2020. \end{sphinxadmonition} \section{7.3 Research Opportunities} \label{\detokenize{cloud:research-opportunities}} \sphinxAtStartPar In order for the scenarios described in this Chapter to become a reality, a wealth of research problems need to be addressed, many of which are a consequence of the blurring line between access networks and the edge cloud. We refer to this as the \sphinxstyleemphasis{access\sphinxhyphen{}edge}, and we conclude by identifying some example challenges/opportunities. \begin{itemize} \item {} \sphinxAtStartPar \sphinxstylestrong{Multi\sphinxhyphen{}Access:} The access\sphinxhyphen{}edge will need to support multiple access technologies (e.g., WiFi, 5G, fiber), and allow users to seamlessly move between them. Research is needed to break down existing technology silos, and design converged solutions to common problems (e.g., security, mobility, QoS). \item {} \sphinxAtStartPar \sphinxstylestrong{Heterogeneity:} Since the access\sphinxhyphen{}edge will be about low\sphinxhyphen{}latency and high\sphinxhyphen{}bandwidth connectivity, much edge functionality will be implemented by programming the forwarding pipeline in white\sphinxhyphen{}box switches, and more generally, will use other domain\sphinxhyphen{}specific processors (e.g., GPUs, TPUs). Research is needed to tailor edge services to take advantage of heterogeneous resources, as well as how to construct end\sphinxhyphen{}to\sphinxhyphen{}end applications from such a collection of building blocks. \item {} \sphinxAtStartPar \sphinxstylestrong{Virtualization:} The access\sphinxhyphen{}edge will virtualize the underlying hardware using a range of techniques, from VMs to containers to serverless functions, interconnected by a range of L2, L3, and L4/7 virtual networks, some of which will be managed by SDN control applications. Research is needed to reconcile the assumptions made about by cloud native services and access\sphinxhyphen{}oriented Virtualized Network Functions (VNFs) about how to virtualize compute, storage, and networking resources. \item {} \sphinxAtStartPar \sphinxstylestrong{Multi\sphinxhyphen{}Tenancy:} The access\sphinxhyphen{}edge will be multi\sphinxhyphen{}tenant, with potentially different stakeholders (operators, service providers, application developers, enterprises) responsible for managing different components. It will not be feasible to run the entire access\sphinxhyphen{}edge in a single trust domain, as different components will operate with different levels of autonomy. Research is needed to minimize the overhead isolation imposes on tenants. \item {} \sphinxAtStartPar \sphinxstylestrong{Customization:} Monetizing the access\sphinxhyphen{}edge will require the ability to offer differentiated and customized services to different classes of subscribers/applications. Sometimes called network slicing (see Section 5.3), this involves support for performance isolation at the granularity of service chains—the sequence of functional elements running on behalf of some subscriber. Research is needed to enforce performance isolation in support of service guarantees. \item {} \sphinxAtStartPar \sphinxstylestrong{Near\sphinxhyphen{}Real Time:} The access\sphinxhyphen{}edge will be a highly dynamic environment, with functionality constantly adapting in response to mobility, workload, and application requirements. Supporting such an environment requires tight control loops, with control software running at the edge. Research is needed to analyze control loops, define analytic\sphinxhyphen{}based controllers, and design dynamically adaptable mechanisms. \item {} \sphinxAtStartPar \sphinxstylestrong{Data Reduction:} The access\sphinxhyphen{}edge will connect an increasing number of devices (not just humans and their handsets), all of which are capable of generating data. Supporting data reduction will be critical, which implies the need for substantial compute capacity (likely including domain\sphinxhyphen{}specific processors) to be available in the access\sphinxhyphen{}edge. Research is needed to refactor applications into their edge\sphinxhyphen{}reduction/backend\sphinxhyphen{}analysis subcomponents. \item {} \sphinxAtStartPar \sphinxstylestrong{Distributed Services:} Services will become inherently distributed, with some aspects running at the access\sphinxhyphen{}edge, some aspects running in the datacenter, and some running on\sphinxhyphen{}premises or in an end device (e.g., on\sphinxhyphen{}vehicle). Supporting such an environment requires a multi\sphinxhyphen{}cloud solution that is decoupled from any single infrastructure\sphinxhyphen{}based platform, with research needed to develop heuristics for function placement. \item {} \sphinxAtStartPar \sphinxstylestrong{Scalability:} The access\sphinxhyphen{}edge will potentially span thousands or even tens of thousands of edge sites. Scaling up the ability to remotely orchestrate that many edge sites (even at just the infrastructure level) will be a qualitatively different challenge than managing a single datacenter. Research is needed to scale both the edge platform and widely deployed edge services. \end{itemize} \phantomsection\label{\detokenize{cloud:reading-democratizing}} \begin{sphinxadmonition}{note}{Further Reading} \sphinxAtStartPar To better understand the research opportunity at the access\sphinxhyphen{}edge, see \sphinxhref{https://ccronline.sigcomm.org/wp-content/uploads/2019/05/acmdl19-289.pdf}{Democratizing the Network Edge}. ACM SIGCOMM CCR, April 2019. \end{sphinxadmonition} \chapter{About The Book} \label{\detokenize{README:about-the-book}}\label{\detokenize{README::doc}} \sphinxAtStartPar Source for \sphinxstyleemphasis{5G Mobile Networks: A Systems Approach} is available on GitHub under terms of the \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{Creative Commons (CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0)} license. The community is invited to contribute corrections, improvements, updates, and new material under the same terms. While this license does not automatically grant the right to make derivative works, we are keen to discuss derivative works (such as translations) with interested parties. Please reach out to \sphinxhref{mailto:discuss@systemsapproach.org}{discuss@systemsapproach.org}. \sphinxAtStartPar If you make use of this work, the attribution should include the following information: \begin{DUlineblock}{0em} \item[] \sphinxstyleemphasis{Title: 5G Mobile Networks: A Systems Approach} \item[] \sphinxstyleemphasis{Authors: Larry Peterson and Oguz Sunay} \item[] \sphinxstyleemphasis{Source:} \sphinxurl{https://github.com/SystemsApproach/5G} \item[] \sphinxstyleemphasis{License:} \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0} \end{DUlineblock} \section{Read the Book} \label{\detokenize{README:read-the-book}} \sphinxAtStartPar This book is part of the \sphinxhref{https://www.systemsapproach.org}{Systems Approach Series}, with an online version published at \sphinxhref{https://5g.systemsapproach.org}{https://5G.systemsapproach.org}. \sphinxAtStartPar To track progress and receive notices about new versions, you can follow the project on \sphinxhref{https://www.facebook.com/Computer-Networks-A-Systems-Approach-110933578952503/}{Facebook} and \sphinxhref{https://twitter.com/SystemsAppr}{Twitter}. To read a running commentary on how the Internet is evolving, follow the \sphinxhref{https://systemsapproach.substack.com}{Systems Approach on Substack}. \section{Build the Book} \label{\detokenize{README:build-the-book}} \sphinxAtStartPar To build a web\sphinxhyphen{}viewable version, you first need to download the source: \begin{sphinxVerbatim}[commandchars=\\\{\}] \PYGZdl{} mkdir \PYGZti{}/systemsapproach \PYGZdl{} cd \PYGZti{}/systemsapproach \PYGZdl{} git clone https://github.com/SystemsApproach/5G.git \PYGZdl{} cd 5G \end{sphinxVerbatim} \sphinxAtStartPar The build process is stored in the Makefile and requires Python be installed. The Makefile will create a virtualenv (\sphinxcode{\sphinxupquote{venv\sphinxhyphen{}docs}}) which installs the documentation generation toolset. You may also need to install the \sphinxcode{\sphinxupquote{enchant}} C library using your system’s package manager for the spelling checker to function properly. \sphinxAtStartPar To generate HTML in \sphinxcode{\sphinxupquote{\_build/html}}, run \sphinxcode{\sphinxupquote{make html}}. \sphinxAtStartPar To check the formatting of the book, run \sphinxcode{\sphinxupquote{make lint}}. \sphinxAtStartPar To check spelling, run \sphinxcode{\sphinxupquote{make spelling}}. If there are additional words, names, or acronyms that are correctly spelled but not in the dictionary, please add them to the \sphinxcode{\sphinxupquote{dict.txt}} file. \sphinxAtStartPar To see the other available output formats, run \sphinxcode{\sphinxupquote{make}}. \section{Contribute to the Book} \label{\detokenize{README:contribute-to-the-book}} \sphinxAtStartPar We hope that if you use this material, you are also willing to contribute back to it. If you are new to open source, you might check out this \sphinxhref{https://opensource.guide/how-to-contribute/}{How to Contribute to Open Source} guide. Among other things, you’ll learn about posting \sphinxstyleemphasis{Issues} that you’d like to see addressed, and issuing \sphinxstyleemphasis{Pull Requests} to merge your improvements back into GitHub. \sphinxAtStartPar If you’d like to contribute and are looking for something that needs attention, see the \sphinxhref{https://github.com/SystemsApproach/5G/wiki}{wiki} for the current TODO list. \chapter{About The Authors} \label{\detokenize{authors:about-the-authors}}\label{\detokenize{authors::doc}} \sphinxAtStartPar \sphinxstylestrong{Larry Peterson} is the Robert E. Kahn Professor of Computer Science, Emeritus at Princeton University, where he served as Chair from 2003\sphinxhyphen{}2009. He is a co\sphinxhyphen{}author of the best selling networking textbook \sphinxstyleemphasis{Computer Networks: A Systems Approach (6e)}, which is now available as open source on GitHub. His research focuses on the design, implementation, and operation of Internet\sphinxhyphen{}scale distributed systems, including the widely used PlanetLab and MeasurementLab platforms. He is currently leading the CORD and Aether access\sphinxhyphen{}edge cloud projects at the Open Networking Foundation (ONF), where he serves CTO. Professor Peterson is a member of the National Academy of Engineering, a Fellow of the ACM and the IEEE, the 2010 recipient of the IEEE Kobayashi Computer and Communication Award, and the 2013 recipient of the ACM SIGCOMM Award. He received his Ph.D. degree from Purdue University in 1985. \sphinxAtStartPar \sphinxstylestrong{Oguz Sunay} is currently the Vice President for Research \& Development at ONF, where he leads all mobile\sphinxhyphen{}related projects. Prior to that, he served as the Chief Architect for Mobile Networking at ONF. Before joining ONF, Sunay was the CTO at Argela\sphinxhyphen{}USA, where he was the innovator of a Programmable Radio Access Network Architecture (ProgRAN) for 5G that enabled the world’s first dynamically programmable RAN slicing solution. He has also held prior industry positions at Nokia Research Center and Bell Laboratories, where he focused on 3G and 4G end\sphinxhyphen{}to\sphinxhyphen{}end systems architectures and participated and chaired various standardization activities. Sunay has also spent over 10 years in academia, as a Professor of Electrical and Computer Engineering. He holds many U.S. and European patents on various aspects of 3G, 4G, and 5G, and has authored numerous journal and conference publications. He received his Ph.D. and M.Sc. from Queen’s University, Canada, and his B.Sc.Hon. from METU, Turkey. \chapter{Read the Latest!} \label{\detokenize{latest:read-the-latest}}\label{\detokenize{latest::doc}} \sphinxAtStartPar \sphinxhref{https://systemsapproach.substack.com/}{Substack Newsletter:} Stay up\sphinxhyphen{}to\sphinxhyphen{}date with the latest developments by following the authors on \sphinxhref{https://systemsapproach.substack.com/}{Substack}, where they connect the concepts and lessons in this book to what’s happening in the Internet today. \sphinxAtStartPar \sphinxhref{https://systemsapproach.org/books/}{Book Series:} Also check out our companion books that cover emerging topics in more depth. \begin{itemize} \item {} \sphinxAtStartPar \sphinxhref{https://sdn.systemsapproach.org}{Software\sphinxhyphen{}Defined Networks: A Systems Approach} \item {} \sphinxAtStartPar \sphinxhref{https://tcpcc.systemsapproach.org}{TCP Congestion Control: A Systems Approach} \item {} \sphinxAtStartPar \sphinxhref{https://ops.systemsapproach.org}{Edge Cloud Operations: A Systems Approach} \end{itemize} \chapter{Print Copies} \label{\detokenize{print:print-copies}}\label{\detokenize{print::doc}} \sphinxAtStartPar We make all books in the \sphinxstyleemphasis{Systems Approach} series available as both print and e\sphinxhyphen{}books. This book is available via Amazon: \sphinxhref{https://amzn.to/3EjwMH0}{5G Mobile Networks: A Systems Approach} \sphinxAtStartPar \sphinxhref{https://systemsapproach.org/books/}{Book Series:} Also check out our companion books that cover networking and emerging topics in more depth. \begin{itemize} \item {} \sphinxAtStartPar \sphinxhref{https://amzn.to/3CtG81U}{Computer Networks: A Systems Approach} \item {} \sphinxAtStartPar \sphinxhref{https://amzn.to/3rmLdCP}{Software\sphinxhyphen{}Defined Networks: A Systems Approach} \item {} \sphinxAtStartPar \sphinxhref{https://amzn.to/3UTYi3T}{TCP Congestion Control: A Systems Approach} \item {} \sphinxAtStartPar \sphinxhref{https://amzn.to/3MfvK13}{Edge Cloud Operations: A Systems Approach} \end{itemize} \sphinxAtStartPar As participants in the Amazon Associate program we may earn income from qualifying purchases using the links above. \renewcommand{\indexname}{Index} \printindex \end{document}