5G-Mobile-Networks-A-Systems-Approach/book.tex

%% Generated by Sphinx.
\def\sphinxdocclass{report}
\documentclass[a4paper,11pt,english]{sphinxmanual}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax
\ifdefined\pdfimageresolution
    \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax
\fi
%% let collapsible pdf bookmarks panel have high depth per default
\PassOptionsToPackage{bookmarksdepth=5}{hyperref}

\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
  \ifdefined\DeclareUnicodeCharacterAsOptional
    \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}}
  \else
    \let\sphinxDUC\DeclareUnicodeCharacter
  \fi
  \sphinxDUC{00A0}{\nobreakspace}
  \sphinxDUC{2500}{\sphinxunichar{2500}}
  \sphinxDUC{2502}{\sphinxunichar{2502}}
  \sphinxDUC{2514}{\sphinxunichar{2514}}
  \sphinxDUC{251C}{\sphinxunichar{251C}}
  \sphinxDUC{2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[LGR,T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}
\usepackage{substitutefont}


\usepackage{tgtermes}
\usepackage{tgheros}
\renewcommand{\ttdefault}{txtt}


\expandafter\ifx\csname T@LGR\endcsname\relax
\else
% LGR was declared as font encoding
  \substitutefont{LGR}{\rmdefault}{cmr}
  \substitutefont{LGR}{\sfdefault}{cmss}
  \substitutefont{LGR}{\ttdefault}{cmtt}
\fi
\expandafter\ifx\csname T@X2\endcsname\relax
  \expandafter\ifx\csname T@T2A\endcsname\relax
  \else
  % T2A was declared as font encoding
    \substitutefont{T2A}{\rmdefault}{cmr}
    \substitutefont{T2A}{\sfdefault}{cmss}
    \substitutefont{T2A}{\ttdefault}{cmtt}
  \fi
\else
% X2 was declared as font encoding
  \substitutefont{X2}{\rmdefault}{cmr}
  \substitutefont{X2}{\sfdefault}{cmss}
  \substitutefont{X2}{\ttdefault}{cmtt}
\fi

\usepackage{textalpha}
\usepackage[Bjarne]{fncychap}
\usepackage[,numfigreset=1,mathnumfig]{sphinx}

\fvset{fontsize=auto}
\usepackage{geometry}


% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}

\addto\captionsenglish{\renewcommand{\contentsname}{Table of Contents}}

\usepackage{sphinxmessages}
\setcounter{tocdepth}{1}



\title{5G Mobile Networks: A Systems Approach}
\date{Dec 09, 2022}
\release{Version 1.1\sphinxhyphen{}dev}
\author{Peterson and Sunay}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{Release}
\makeindex
\begin{document}

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}



\chapter*{Preface}
\label{\detokenize{preface:preface}}\label{\detokenize{preface::doc}}
\sphinxAtStartPar
The transition to 5G is happening, and unless you’ve been actively
trying to ignore it, you’ve undoubtedly heard the hype. But if you are
like 99\% of the CS\sphinxhyphen{}trained, systems\sphinxhyphen{}oriented, cloud\sphinxhyphen{}savvy people in
the world, the cellular network is largely a mystery. You know it’s an
important technology used in the last mile to connect people to the
Internet, but you’ve otherwise abstracted it out of your
scope\sphinxhyphen{}of\sphinxhyphen{}concerns. Perhaps you’ve heard a conspiracy theory or two
about how 5G is going to cause some awful side\sphinxhyphen{}effect—a sign that the
hype around 5G might not be working as intended.

\sphinxAtStartPar
The important thing to understand about 5G is that it implies much more
than a generational upgrade in bandwidth (although it does involve
that too). It involves transformative
changes to the access network that will make it much more like a
modern cloud. Important technology trends such as software\sphinxhyphen{}defined
networking (SDN) and open source software will lead to an
access network that is much more nimble and innovative. And
it will enable new classes of application, particularly a broad set of
IoT (Internet of Things) applications. In fact we could even see the
“Access\sphinxhyphen{}as\sphinxhyphen{}frontend\sphinxhyphen{}to\sphinxhyphen{}Internet” perspective being turned on its head, as the
access network itself becomes a collection of clouds with new services
delivered directly from “edge clouds”.

\sphinxAtStartPar
This book is written for someone who has a working understanding of the
Internet and the cloud, but has had limited success penetrating the myriad
of acronyms that dominate cellular networking. In fairness, the Internet
has its share of acronyms, but it also comes with a sufficient set of
abstractions to help manage the complexity. It’s hard to say the same
for the cellular network, where pulling on one thread seemingly unravels
the entire space. It has also been the case that the cellular network
had been largely hidden inside proprietary devices, which has made it
impossible to figure it out for yourself.

\sphinxAtStartPar
This book is the result of a mobile networking expert teaching a
systems person about 5G as we’ve collaborated on an open source 5G
implementation. The material has been used to train other software
developers, and we are hopeful it will be useful to anyone who wants a
deeper understanding of 5G and the opportunity for innovation it
provides. Readers who want hands\sphinxhyphen{}on experience can also access the open
source software introduced in the book.

\sphinxAtStartPar
This book will likely be a work\sphinxhyphen{}in\sphinxhyphen{}progress for the foreseeable future.
It’s not intended to be encyclopedic—favoring perspective and end\sphinxhyphen{}to\sphinxhyphen{}end
completeness over every last bit of detail—but we do plan to flesh out
the content over time. Your suggestions (and contributions) to this end
are welcome.


\section*{Acknowledgements}
\label{\detokenize{preface:acknowledgements}}
\sphinxAtStartPar
The software described in this book is due to the hard work of the ONF
engineering team and the open source community that works with
them. We acknowledge their contributions. We also thank Bruce Davie,
Guru Parulkar, and Changhoon Kim for their feedback on early drafts of
the manuscript.

\begin{DUlineblock}{0em}
\item[] Larry Peterson and Oguz Sunay
\item[] Open Networking Foundation
\item[] March 2020
\end{DUlineblock}


\chapter{Chapter 1:  Introduction}
\label{\detokenize{intro:chapter-1-introduction}}\label{\detokenize{intro::doc}}
\sphinxAtStartPar
Mobile networks, which have a 40\sphinxhyphen{}year history that parallels the
Internet’s, have undergone significant change. The first two
generations supported voice and then text, with 3G defining the
transition to broadband access, supporting data rates measured in
hundreds of kilobits\sphinxhyphen{}per\sphinxhyphen{}second. Today, the industry is at 4G
(supporting data rates typically measured in the few
megabits\sphinxhyphen{}per\sphinxhyphen{}second) and transitioning to 5G, with the promise of a
tenfold increase in data rates.

\sphinxAtStartPar
But 5G is about much more than increased bandwidth. 5G represents a
fundamental rearchitecting of the access network in a way that
leverages several key technology trends and sets it on a path to enable
much greater innovation. In the same way that 3G
defined the transition from voice to broadband, 5G’s promise is primarily
about the transition from a single access service (broadband
connectivity) to a richer collection of edge services and devices. 5G
is expected to provide support for immersive user interfaces (e.g., AR/VR),
mission\sphinxhyphen{}critical applications (e.g., public safety, autonomous
vehicles), and the Internet\sphinxhyphen{}of\sphinxhyphen{}Things (IoT). Because these use cases
will include everything from home appliances to industrial robots to
self\sphinxhyphen{}driving cars, 5G won’t just support humans accessing the Internet
from their smartphones, but also swarms of autonomous devices working
together on their behalf. There is more to supporting these services
than just improving bandwidth or latency to individual users. As we will see, a fundamentally
different edge network architecture is required.

\sphinxAtStartPar
The requirements for this architecture are ambitious, and can be
illustrated by three classes of capabilities:
\begin{itemize}
\item {} 
\sphinxAtStartPar
To support \sphinxstyleemphasis{Massive Internet\sphinxhyphen{}of\sphinxhyphen{}Things}, potentially including
devices with ultra\sphinxhyphen{}low energy (10+ years of battery life), ultra\sphinxhyphen{}low
complexity (10s of bits\sphinxhyphen{}per\sphinxhyphen{}second), and ultra\sphinxhyphen{}high density (1
million nodes per square kilometer).

\item {} 
\sphinxAtStartPar
To support \sphinxstyleemphasis{Mission\sphinxhyphen{}Critical Control}, potentially including
ultra\sphinxhyphen{}high availability (greater than 99.999\% or “five nines”),
ultra\sphinxhyphen{}low latency (as low as 1 ms), and extreme mobility (up to 100
km/h).

\item {} 
\sphinxAtStartPar
To support \sphinxstyleemphasis{Enhanced Mobile Broadband}, potentially including extreme data rates
(multi\sphinxhyphen{}Gbps peak, 100+ Mbps sustained) and extreme
capacity (10 Tbps of aggregate throughput per square kilometer).

\end{itemize}

\sphinxAtStartPar
These targets will certainly not be met overnight, but that’s in keeping
with each generation of the mobile network being a decade\sphinxhyphen{}long
endeavor.

\sphinxAtStartPar
On top of these quantitative improvements to the capabilities of the access
network, 5G is being viewed as a chance for building a platform to
support innovation. Whereas prior access networks were generally
optimized for known services (such as voice calls and SMS), the
Internet has been hugely successful in large part because it supported
a wide range of applications that were not even thought of when it was
first designed. The 5G network is very much being designed with this
same goal of enabling all sorts of future applications beyond those we
fully recognize today.

\phantomsection\label{\detokenize{intro:reading-vision}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For an example of the grand vision for 5G from one of the industry
leaders, see \sphinxhref{https://www.qualcomm.com/media/documents/files/whitepaper-making-5g-nr-a-reality.pdf}{Making 5G NR a Reality}.
Qualcomm Whitepaper, December 2016.
\end{sphinxadmonition}

\sphinxAtStartPar
The 5G mobile network, because it is on an
evolutionary path and not a point solution, includes standardized
specifications, a range of implementation choices, and a long list of
aspirational goals. Because this leaves so much room for interpretation,
our approach to describing 5G is grounded in two mutually supportive
principles. The first is to apply a \sphinxstyleemphasis{systems lens}, which is to say, we
explain the sequence of design decisions that lead to a solution rather
than fall back on enumerating the overwhelming number of acronyms or
individual point technologies as a
\sphinxstyleemphasis{fait accompli}. The second is to aggressively disaggregate the system.
Building a disaggregated, virtualized, and software\sphinxhyphen{}defined 5G access
network is the direction the industry is already headed (for good
technical and business reasons), but breaking the 5G network down into
its elemental components is also the best way to explain how 5G works.
It also helps to illustrate how 5G might evolve in the future to provide
even more value.

\begin{sphinxShadowBox}
\sphinxstylesidebartitle{Evolutionary Path}

\sphinxAtStartPar
That 5G is on an evolutionary path is the central theme of
this book.  We call attention to its importance here, and
revisit the topic throughout the book.

\sphinxAtStartPar
We are writing this book for \sphinxstyleemphasis{system generalists}, with the
goal of helping bring a community that understands a broad
range of systems issues (but knows little or nothing about the
cellular network) up to speed so they can play a role in its
evolution. This is a community that understands both feature
velocity and best practices in building robust scalable
systems, and so has an important role to play in bringing all
of 5G’s potential to fruition.
\end{sphinxShadowBox}

\sphinxAtStartPar
What this all means is that there is no single, comprehensive definition
of 5G, any more than there is for the Internet. It is a complex and
evolving system, constrained by a set of standards that purposely give
all the stakeholders many degrees of freedom. In the chapters that
follow, it should be clear from the context whether we are talking about
\sphinxstyleemphasis{standards} (what everyone must do to interoperate), \sphinxstyleemphasis{trends} (where
the industry seems to be headed), or \sphinxstyleemphasis{implementation choices}
(examples to make the discussion more concrete). By adopting a systems
perspective throughout, our intent is to describe 5G in a way that
helps the reader navigate this rich and rapidly evolving system.


\section{1.1 Standardization Landscape}
\label{\detokenize{intro:standardization-landscape}}
\sphinxAtStartPar
As of 3G, the generational designation corresponds to a standard defined
by the \sphinxstyleemphasis{3rd Generation Partnership Project (3GPP)}. Even though its name
has “3G” in it, the 3GPP continues to define the standards for 4G and 5G,
each of which corresponds to a sequence of releases of the standard.
Release 15 is considered the demarcation point between 4G and 5G, with
Release 17 scheduled for 2021. Complicating the terminology, 4G
was on a multi\sphinxhyphen{}release evolutionary path referred to as \sphinxstyleemphasis{Long Term
Evolution (LTE)}. 5G is on a similar evolutionary path, with several
expected releases over its lifetime.

\sphinxAtStartPar
While 5G is an ambitious advance beyond 4G, it is also the case that
understanding 4G is the first step to understanding 5G, as several
aspects of the latter can be explained as bringing a new
degree\sphinxhyphen{}of\sphinxhyphen{}freedom to the former. In the chapters that follow, we often
introduce some architectural feature of 4G as a way of laying the
foundation for the corresponding 5G component.

\sphinxAtStartPar
Like Wi\sphinxhyphen{}Fi, cellular networks transmit data at certain bandwidths in the
radio spectrum. Unlike Wi\sphinxhyphen{}Fi, which permits anyone to use a channel at
either 2.4 or 5 GHz (these are unlicensed bands), governments have
auctioned off and licensed exclusive use of various frequency bands to
service providers, who in turn sell mobile access service to their
subscribers.

\sphinxAtStartPar
There is also a shared\sphinxhyphen{}license band at 3.5 GHz, called \sphinxstyleemphasis{Citizens
Broadband Radio Service (CBRS)}, set aside in North America for cellular
use. Similar spectrum is being set aside in other countries. The CBRS band
allows 3 tiers of users to share the spectrum: first right of use
goes to the original owners of this spectrum (naval radars and satellite
ground stations); followed by priority users who receive this right over
10MHz bands for three years via regional auctions; and finally the rest
of the population, who can access and utilize a portion of this band as
long as they first check with a central database of registered users.
CBRS, along with standardization efforts to extend cellular networks to
operate in the unlicensed bands, open the door for private cellular
networks similar to Wi\sphinxhyphen{}Fi.

\sphinxAtStartPar
The specific frequency bands that are licensed for cellular networks
vary around the world, and are complicated by the fact that network
operators often simultaneously support both old/legacy technologies and
new/next\sphinxhyphen{}generation technologies, each of which occupies a different
frequency band. The high\sphinxhyphen{}level summary is that traditional cellular
technologies range from 700\sphinxhyphen{}2400 MHz, with new mid\sphinxhyphen{}spectrum
allocations now happening at 6 GHz, and millimeter\sphinxhyphen{}wave (mmWave)
allocations opening above 24 GHz.

\sphinxAtStartPar
While the specific frequency band is not directly relevant to
understanding 5G from an architectural perspective, it does impact the
physical\sphinxhyphen{}layer components, which in turn has indirect ramifications on
the overall 5G system. We identify and explain these ramifications
in later chapters. Ensuring that the allocated spectrum is used
\sphinxstyleemphasis{efficiently} is also a critical design goal.


\section{1.2 Access Networks}
\label{\detokenize{intro:access-networks}}
\sphinxAtStartPar
The cellular network is part of the access network that implements the
Internet’s so\sphinxhyphen{}called \sphinxstyleemphasis{last mile}. Other access technologies include
\sphinxstyleemphasis{Passive Optical Networks (PON)}, colloquially known as
Fiber\sphinxhyphen{}to\sphinxhyphen{}the\sphinxhyphen{}Home. These access networks are provided by both big and
small network operators. Global network operators like AT\&T run access
networks at thousands of aggregation points\sphinxhyphen{}of\sphinxhyphen{}presence across a
country like the US, along with a national backbone that interconnects
those sites. Small regional and municipal network operators might run
an access network with one or two points\sphinxhyphen{}of\sphinxhyphen{}presence, and then connect
to the rest of the Internet through some large operator’s backbone.

\sphinxAtStartPar
In either case, access networks are physically anchored at thousands of
aggregation points\sphinxhyphen{}of\sphinxhyphen{}presence within close proximity to end users,
each of which serves anywhere from 1,000\sphinxhyphen{}100,000 subscribers,
depending on population density. In practice, the physical deployment
of these “edge” locations vary from operator to operator, but one
possible scenario is to anchor both the cellular and wireline access
networks in Telco \sphinxstyleemphasis{Central Offices}.

\sphinxAtStartPar
Historically, the Central Office—officially known as the \sphinxstyleemphasis{PSTN
(Public Switched Telephone Network) Central Office}—anchored wired
access (both telephony and broadband), while the cellular network
evolved independently by deploying a parallel set of \sphinxstyleemphasis{Mobile Telephone
Switching Offices (MTSO)}. Each MTSO serves as a \sphinxstyleemphasis{mobile aggregation}
point for the set of cell towers in a given geographic area. For our
purposes, the important idea is that such aggregation points exist, and
it is reasonable to think of them as defining the edge of the
operator\sphinxhyphen{}managed access network. For simplicity, we sometimes use the
term “Central Office” as a synonym for both types of edge sites.


\section{1.3 Edge Cloud}
\label{\detokenize{intro:edge-cloud}}
\sphinxAtStartPar
Because of their wide distribution and close proximity to end users,
Central Offices are also an ideal place to host the edge cloud. But this
raises the question: What exactly is the edge cloud?

\sphinxAtStartPar
In a nutshell, the cloud began as a collection of warehouse\sphinxhyphen{}sized
datacenters, each of which provided a cost\sphinxhyphen{}effective way to power, cool,
and operate a scalable number of servers. Over time, this shared
infrastructure lowered the barrier to deploying scalable Internet
services, but today, there is increasing pressure to offer
low\sphinxhyphen{}latency/high\sphinxhyphen{}bandwidth cloud applications that cannot be effectively
implemented in centralized datacenters. Augmented Reality (AR), Virtual
Reality (VR), Internet\sphinxhyphen{}of\sphinxhyphen{}Things (IoT), and Autonomous Vehicles are all
examples of this kind of application. This has resulted in a trend to
move some functionality out of the datacenter and towards the edge of
the network, closer to end users.

\sphinxAtStartPar
Where this edge is \sphinxstyleemphasis{physically} located depends on who you ask. If you
ask a network operator that already owns and operates thousands of
Central Offices, then their Central Offices are an obvious answer.
Others might claim the edge is located at the 14,000 Starbucks across
the US, and still others might point to the tens\sphinxhyphen{}of\sphinxhyphen{}thousands of cell
towers spread across the globe.

\sphinxAtStartPar
Our approach is to be location agnostic, but it is worth pointing out
that the cloud’s migration to the edge coincides with a second trend,
which is that network operators are re\sphinxhyphen{}architecting the access network
to use the same commodity hardware and best practices in building
scalable software as the cloud providers. Such a design, which is
sometimes referred to as CORD \sphinxstyleemphasis{(Central Office Re\sphinxhyphen{}architected as a
Datacenter)}, supports both the access network and edge services
co\sphinxhyphen{}located on a shared cloud platform. This platform is then replicated
across hundreds or thousands of sites (including, but not limited to,
Central Offices). So while we shouldn’t limit ourselves to the Central
Office as the only answer to the question of where the edge cloud is
located, it is becoming a viable option.

\phantomsection\label{\detokenize{intro:reading-cord}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
To learn about the technical origins of CORD, which was first
applied to fiber\sphinxhyphen{}based access networks (PON), see \sphinxhref{https://wiki.opencord.org/download/attachments/1278027/PETERSON\_CORD.pdf}{Central Office
Re\sphinxhyphen{}architected as a Datacenter, IEEE Communications, October 2016}.

\sphinxAtStartPar
To understand the business case for CORD (and CORD\sphinxhyphen{}inspired
technologies), see the A.D. Little report \sphinxhref{https://www.adlittle.com/en/who-dares-wins}{Who Dares Wins!
How Access Transformation Can Fast\sphinxhyphen{}Track Evolution of
Operator Production Platforms, September 2019}.
\end{sphinxadmonition}

\sphinxAtStartPar
When we get into the details of how 5G can be implemented in practice,
we use CORD as our exemplar. For now, the important thing to understand
is that 5G is being implemented as software running on commodity
hardware, rather than embedded in the special\sphinxhyphen{}purpose proprietary
hardware used in past generations. This has a significant impact on how
we think about 5G (and how we describe 5G), which will increasingly
become yet another software\sphinxhyphen{}based component in the cloud, as opposed to
an isolated and specialized technology attached to the periphery of the
cloud.

\sphinxAtStartPar
Keep in mind that our use of CORD as an exemplar is not to imply that
the edge cloud is limited to Central Offices. CORD is a good exemplar
because it is designed to host both edge services and access
technologies like 5G on a common platform, where the Telco Central
Office is one possible location to deploy such a platform.

\sphinxAtStartPar
An important takeaway from this discussion is that to understand how 5G
is being implemented, it is helpful to have a working understanding of
how clouds are built. This includes the use of \sphinxstyleemphasis{commodity hardware}
(both servers and white\sphinxhyphen{}box switches), horizontally scalable
\sphinxstyleemphasis{microservices} (also referred to as \sphinxstyleemphasis{cloud native}), and
\sphinxstyleemphasis{Software\sphinxhyphen{}Defined Networks (SDN)}. It is also helpful to have an
appreciation for how cloud software is developed, tested, deployed, and
operated, including practices like \sphinxstyleemphasis{DevOps} and \sphinxstyleemphasis{Continuous Integration
/ Continuous Deployment (CI/CD)}.

\phantomsection\label{\detokenize{intro:reading-devops}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
If you are unfamiliar with SDN, we recommend a companion book:
\sphinxhref{https://sdn.systemsapproach.org/}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. March 2020.

\sphinxAtStartPar
If you are unfamiliar with DevOps—or more generally, with the
operational issues cloud providers face—we recommend \sphinxhref{https://landing.google.com/sre/books/}{Site
Reliability Engineering: How Google Runs Production Systems}.
\end{sphinxadmonition}

\sphinxAtStartPar
One final note about terminology. Anyone who has been paying
attention to the discussion surrounding 5G will have undoubtedly heard
about \sphinxstyleemphasis{Network Function Virtualization (NFV)}, which involves moving
functionality that was once embedded in hardware appliances into VMs
(or, more recently, containers)
running on commodity servers. In our experience, NFV is a stepping
stone towards the fully disaggregated and cloud native solution we
describe in this book, and so we do not dwell on it. You can think of
the NFV initiative as mostly consistent with the approach taken in
this book, but making some specific engineering choices that may
differ in detail from that described here.

\sphinxAtStartPar
While equating NFV with an implementation choice is perfectly valid,
there is another interpretation of events that better captures the
essence of the transformation currently underway. When Telcos began
the NFV initiative, they imagined incorporating cloud technologies
into their networks, creating a so\sphinxhyphen{}called \sphinxstyleemphasis{Telco Cloud}.  What is
actually happening instead, is that the Telco’s access technology is
being subsumed into the cloud, running as yet another cloud\sphinxhyphen{}hosted
workload. It would be more accurate to refer to the resulting system
now emerging as the \sphinxstyleemphasis{Cloud\sphinxhyphen{}based Telco}. One reading of this book is
as a roadmap to such an outcome.


\chapter{Chapter 2:  Radio Transmission}
\label{\detokenize{primer:chapter-2-radio-transmission}}\label{\detokenize{primer::doc}}
\sphinxAtStartPar
For anyone familiar with wireless access technologies like Wi\sphinxhyphen{}Fi, the
cellular network is most unique due to its approach to sharing the
available radio spectrum among its many users, all the while allowing
those users to remain connected while moving. This has resulted in a
highly dynamic and adaptive approach, in which coding, modulation and
scheduling play a central role.

\sphinxAtStartPar
As we will see in this chapter, cellular networks use a
reservation\sphinxhyphen{}based strategy, whereas Wi\sphinxhyphen{}Fi is contention\sphinxhyphen{}based. This
difference is rooted in each system’s fundamental assumption about
utilization: Wi\sphinxhyphen{}Fi assumes a lightly loaded network (and hence
optimistically transmits when the wireless link is idle and backs off if
contention is detected), while 4G and 5G cellular networks assume (and
strive for) high utilization (and hence explicitly assign different
users to different “shares” of the available radio spectrum).

\sphinxAtStartPar
We start by giving a short primer on radio transmission as a way of
laying a foundation for understanding the rest of the 5G architecture.
The following is not a substitute for a theoretical treatment of the topic,
but is instead intended as a way of grounding the systems\sphinxhyphen{}oriented
description of 5G that follows in the reality of wireless communication.


\section{2.1 Coding and Modulation}
\label{\detokenize{primer:coding-and-modulation}}
\sphinxAtStartPar
The mobile channel over which digital data needs to be reliably
transmitted brings a number of impairments, including noise,
attenuation, distortion, fading, and interference. This challenge is
addressed by a combination of coding and modulation, as depicted in
\hyperref[\detokenize{primer:fig-modulation}]{Figure \ref{\detokenize{primer:fig-modulation}}}.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide09}.png}
\caption{The role of coding and modulation in mobile communication.}\label{\detokenize{primer:id5}}\label{\detokenize{primer:fig-modulation}}\end{figure}

\sphinxAtStartPar
At its core, coding inserts extra bits into the data to help recover
from all the environmental factors that interfere with signal
propagation. This typically implies some form of \sphinxstyleemphasis{Forward Error
Correction} (e.g., turbo codes, polar codes). Modulation then
generates signals that represent the encoded data stream, and it does
so in a way that matches the channel characteristics: It first uses a
digital modulation signal format that maximizes the number of reliably
transmitted bits every second based on the specifics of the observed
channel impairments; it next matches the transmission
bandwidth to channel bandwidth using pulse shaping; and finally, it
uses RF modulation to transmit the signal as an electromagnetic wave
over an assigned \sphinxstyleemphasis{carrier frequency.}

\sphinxAtStartPar
For a deeper appreciation of the challenges of reliably transmitting
data by propagating radio signals through the air, consider the
scenario depicted in \hyperref[\detokenize{primer:fig-multipath}]{Figure \ref{\detokenize{primer:fig-multipath}}}, where
the signal bounces off various stationary and moving objects,
following multiple paths from the transmitter to the receiver, who may
also be moving.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide10}.png}
\caption{Signals propagate along multiple paths from
transmitter to receiver.}\label{\detokenize{primer:id6}}\label{\detokenize{primer:fig-multipath}}\end{figure}

\sphinxAtStartPar
As a consequence of these multiple paths, the original signal arrives at
the receiver spread over time, as illustrated in
\hyperref[\detokenize{primer:fig-coherence}]{Figure \ref{\detokenize{primer:fig-coherence}}}. Empirical evidence shows that the
Multipath Spread—the time between the first and last signals of one
transmission arriving at the receiver—is 1\sphinxhyphen{}10μs in urban
environments and 10\sphinxhyphen{}30μs in suburban environments. These multipath
signals can interfere with each other constructively or destructively,
and this will vary over time. Theoretical
bounds for the time duration for which the channel may be assumed to
be invariant, known as the \sphinxstyleemphasis{Coherence Time} and denoted
\(T_c\), is given by
\begin{equation*}
\begin{split}T_c =c/v \times 1/f\end{split}
\end{equation*}
\sphinxAtStartPar
where \(c\) is the velocity of the signal, \(v\) is the
velocity of the receiver (e.g., moving car or train), and \(f\) is
the frequency of the carrier signal that is being modulated. This
says the coherence time is inversely proportional to the frequency of
the signal and the speed of movement, which makes intuitive sense: The
higher the frequency (narrower the wave) the shorter the coherence time,
and likewise, the faster the receiver is moving the shorter the coherence
time. Based on the target parameters to this model (selected according
to the target physical environment), it is possible to calculate
\(T_c\), which in turn bounds the rate at which symbols can be
transmitted without undue risk of interference. The dynamic nature of
the wireless channel is a central challenge to address in the cellular
network.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide11}.png}
\caption{Received data spread over time due to multipath
variation.}\label{\detokenize{primer:id7}}\label{\detokenize{primer:fig-coherence}}\end{figure}

\sphinxAtStartPar
To complicate matters further,
\hyperref[\detokenize{primer:fig-multipath}]{Figure \ref{\detokenize{primer:fig-multipath}}} and \hyperref[\detokenize{primer:fig-coherence}]{\ref{\detokenize{primer:fig-coherence}}} imply
the transmission originates from a single
antenna, but cell towers are equipped with an array of antennas, each
transmitting in a different (but overlapping) direction. This
technology, called \sphinxstyleemphasis{Multiple\sphinxhyphen{}Input\sphinxhyphen{}Multiple\sphinxhyphen{}Output (MIMO)}, opens the
door to purposely transmitting data from multiple antennas in an effort
to reach the receiver, adding even more paths to the environment\sphinxhyphen{}imposed
multipath propagation.

\sphinxAtStartPar
One of the most important consequences of these factors is that the
transmitter must receive feedback from every receiver to judge how to
best utilize the wireless medium on their behalf. 3GPP specifies a
\sphinxstyleemphasis{Channel Quality Indicator (CQI)} for this purpose, where in practice
the receiver sends a CQI status report to the base station periodically
(e.g., every millisecond in LTE). These CQI messages report the observed
signal\sphinxhyphen{}to\sphinxhyphen{}noise ratio, which impacts the receiver’s ability to recover
the data bits. The base station then uses this information to adapt how
it allocates the available radio spectrum to the subscribers it is
serving, as well as which coding and modulation scheme to employ.
All of these decisions are made by the scheduler.


\section{2.2 Scheduler}
\label{\detokenize{primer:scheduler}}
\sphinxAtStartPar
How the scheduler does its job is one of the most important properties
of each generation of the cellular network, which in turn depends on
the multiplexing mechanism. For example, 2G used \sphinxstyleemphasis{Time Division
Multiple Access (TDMA)} and 3G used \sphinxstyleemphasis{Code Division Multiple Access
(CDMA)}. How data is multiplexed is also a major differentiator for 4G
and 5G, completing the transition from the cellular network being
fundamentally circuit\sphinxhyphen{}switched to fundamentally packet\sphinxhyphen{}switched.

\sphinxAtStartPar
Both 4G and 5G are based on \sphinxstyleemphasis{Orthogonal Frequency\sphinxhyphen{}Division
Multiplexing (OFDM)}, an approach that multiplexes data over multiple
orthogonal subcarrier frequencies, each of which is modulated
independently. The value and efficiency of OFDM is in how it selects
subcarrier frequencies so as to avoid interference, that is, how it
achieves orthogonality. That topic is beyond the scope of this book.
We instead take a decidedly abstract perspective of multiplexing,
focusing on “discrete schedulable units of the radio spectrum” rather
than the signalling and modulation underpinnings that yield those
schedulable units.

\sphinxAtStartPar
To start, we drill down on these schedulable units. We return to the
broader issue of the \sphinxstyleemphasis{air interface} that makes efficient use of the
spectrum in the concluding section.


\subsection{Multiplexing in 4G}
\label{\detokenize{primer:multiplexing-in-4g}}
\sphinxAtStartPar
The 4G approach to multiplexing downstream transmissions is called
\sphinxstyleemphasis{Orthogonal Frequency\sphinxhyphen{}Division Multiple Access (OFDMA)}, a specific
application of OFDM that multiplexes data over a set of 12 orthogonal
subcarrier frequencies, each of which is modulated independently.%
\begin{footnote}[1]\sphinxAtStartFootnote
4G uses a different multiplexing strategy for upstream
transmissions (from user devices to base stations), but we do
not describe it because the approach is not applicable to 5G.
%
\end{footnote} The
“Multiple Access” in OFDMA implies that data can simultaneously be
sent on behalf of multiple users, each on a different subcarrier
frequency and for a different duration of time. The subbands are
narrow (e.g., 15 kHz), but the coding of user data into OFDMA symbols
is designed to minimize the risk of data loss due to interference
between adjacent bands.

\sphinxAtStartPar
The use of OFDMA naturally leads to conceptualizing the radio spectrum
as a 2\sphinxhyphen{}D resource, as shown in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}.
The minimal schedulable unit, called a \sphinxstyleemphasis{Resource Element (RE)},
corresponds to a 15\sphinxhyphen{}kHz band around one subcarrier frequency and the
time it takes to transmit one OFDMA symbol. The number of bits that can
be encoded in each symbol depends on the modulation rate, so for example
using \sphinxstyleemphasis{Quadrature Amplitude Modulation (QAM)}, 16\sphinxhyphen{}QAM yields 4 bits per
symbol and 64\sphinxhyphen{}QAM yields 6 bits per symbol.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide12}.png}
\caption{Spectrum abstractly represented by a 2\sphinxhyphen{}D grid of
schedulable Resource Elements.}\label{\detokenize{primer:id8}}\label{\detokenize{primer:fig-sched-grid}}\end{figure}

\sphinxAtStartPar
A scheduler allocates some number of REs to each user that has data to
transmit during each 1 ms \sphinxstyleemphasis{Transmission Time Interval (TTI)}, where users
are depicted by different colored blocks in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}.
The only constraint on the scheduler is that it must make its allocation
decisions on blocks of 7x12=84 resource elements, called a \sphinxstyleemphasis{Physical
Resource Block (PRB)}. \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}} shows two
back\sphinxhyphen{}to\sphinxhyphen{}back PRBs. Of course time continues to flow along one axis, and
depending on the size of the available frequency band (e.g., it might be
100 MHz wide), there may be many more subcarrier slots (and hence PRBs)
available along the other axis, so the scheduler is essentially
preparing and transmitting a sequence of PRBs.

\sphinxAtStartPar
Note that OFDMA is not a coding/modulation algorithm, but instead
provides a framework for selecting a specific coding and modulator for
each subcarrier frequency. QAM is one common example modulator. It is
the scheduler’s responsibility to select the modulation to use for each
PRB, based on the CQI feedback it has received. The scheduler also
selects the coding on a per\sphinxhyphen{}PRB basis, for example, by how it sets the
parameters to the turbo code algorithm.

\sphinxAtStartPar
The 1\sphinxhyphen{}ms TTI corresponds to the time frame in which the scheduler
receives feedback from users about the quality of the signal they are
experiencing. This is the CQI mentioned earlier, where once every
millisecond, each user sends a set of metrics, which the scheduler uses
to make its decision as to how to allocate PRBs during the subsequent
TTI.

\sphinxAtStartPar
Another input to the scheduling decision is the \sphinxstyleemphasis{QoS Class Identifier
(QCI)}, which indicates the quality\sphinxhyphen{}of\sphinxhyphen{}service each class of traffic is
to receive. In 4G, the QCI value assigned to each class (there are nine
such classes, in total) indicates whether the traffic has a \sphinxstyleemphasis{Guaranteed
Bit Rate (GBR)} or not \sphinxstyleemphasis{(non\sphinxhyphen{}GBR)}, plus the class’s relative priority
within those two categories.

\sphinxAtStartPar
Finally, keep in mind that \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}} focuses on
scheduling transmissions from a single antenna, but the MIMO technology
described above means the scheduler also has to determine which antenna
(or more generally, what subset of antennas) will most effectively reach
each receiver. But again, in the abstract, the scheduler is charged with
allocating a sequence of Resource Elements.

\sphinxAtStartPar
This all raises the question: How does the scheduler decide which set of
users to service during a given time interval, how many resource
elements to allocate to each such user, how to select the coding and
modulation levels, and which antenna to transmit their data on? This is
an optimization problem that, fortunately, we are not trying to solve
here. Our goal is to describe an architecture that allows someone else
to design and plug in an effective scheduler. Keeping the cellular
architecture open to innovations like this is one of our goals, and as
we will see in the next section, becomes even more important in 5G where
the scheduler operates with even more degrees of freedom.


\subsection{Multiplexing in 5G}
\label{\detokenize{primer:multiplexing-in-5g}}
\sphinxAtStartPar
The transition from 4G to 5G introduces additional flexibility in
how the radio spectrum is scheduled, making it possible to adapt the
cellular network to a more diverse set of devices and applications
domains.

\sphinxAtStartPar
Fundamentally, 5G defines a family of waveforms—unlike LTE, which
specified only one waveform—each optimized for a different band in the
radio spectrum.%
\begin{footnote}[2]\sphinxAtStartFootnote
A waveform is the frequency, amplitude, and phase\sphinxhyphen{}shift
independent property (shape) of a signal. A sine wave is an example
waveform.
%
\end{footnote}  The bands with carrier frequencies below 1 GHz are
designed to deliver mobile broadband and massive IoT services with a
primary focus on range. Carrier frequencies between 1\sphinxhyphen{}6 GHz are
designed to offer wider bandwidths, focusing on mobile broadband and
mission\sphinxhyphen{}critical applications. Carrier frequencies above 24 GHz
(mmWaves) are designed to provide super wide bandwidths over short,
line\sphinxhyphen{}of\sphinxhyphen{}sight coverage.

\sphinxAtStartPar
These different waveforms affect the scheduling and subcarrier intervals
(i.e., the “size” of the resource elements described in the previous
section).
\begin{itemize}
\item {} 
\sphinxAtStartPar
For sub\sphinxhyphen{}1 GHz bands, 5G allows maximum 50 MHz bandwidths. In this case,
there are two waveforms: one with subcarrier spacing of 15 kHz and
another of 30 kHz. (We used 15 kHz in the example shown in
\hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}.)
The corresponding scheduling intervals are
0.5 and 0.25 ms, respectively. (We used 0.5 ms in the example shown
in \hyperref[\detokenize{primer:fig-sched-grid}]{Figure \ref{\detokenize{primer:fig-sched-grid}}}.)

\item {} 
\sphinxAtStartPar
For 1\sphinxhyphen{}6 GHz bands, maximum bandwidths go up to 100 MHz.
Correspondingly, there are three waveforms with subcarrier spacings
of 15, 30 and 60 kHz, corresponding to scheduling intervals of
0.5, 0.25, and 0.125 ms, respectively.

\item {} 
\sphinxAtStartPar
For millimeter bands, bandwidths may go up to 400 MHz. There are two
waveforms, with subcarrier spacings of 60 kHz and 120 kHz. Both have
scheduling intervals of 0.125 ms.

\end{itemize}

\sphinxAtStartPar
These various configurations of subcarrier spacing and scheduling
intervals are sometimes called the \sphinxstyleemphasis{numerology} of the radio’s air
interface.

\sphinxAtStartPar
This range of numerology is important because it adds another degree
of freedom to the scheduler. In addition to allocating radio resources
to users, it has the ability to dynamically adjust the size of the
resource by changing the wave form being used. With this additional
freedom, fixed\sphinxhyphen{}sized REs are no longer the primary unit of resource
allocation.  We instead use more abstract terminology, and talk about
allocating \sphinxstyleemphasis{Resource Blocks} to subscribers, where the 5G scheduler
determines both the size and number of Resource Blocks allocated
during each time interval.

\sphinxAtStartPar
\hyperref[\detokenize{primer:fig-scheduler}]{Figure \ref{\detokenize{primer:fig-scheduler}}} depicts the role of the scheduler
from this more abstract perspective, where just as with 4G, CQI
feedback from the receivers and the QCI quality\sphinxhyphen{}of\sphinxhyphen{}service class
selected by the subscriber are the two key pieces of input to the
scheduler. Note that the set of QCI values changes between 4G and 5G,
reflecting the increasing differentiation being supported. For 5G,
each class includes the following attributes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Resource Type: Guaranteed Bit Rate (GBR), Delay\sphinxhyphen{}Critical GBR, Non\sphinxhyphen{}GBR

\item {} 
\sphinxAtStartPar
Priority Level

\item {} 
\sphinxAtStartPar
Packet Delay Budget

\item {} 
\sphinxAtStartPar
Packet Error Rate

\item {} 
\sphinxAtStartPar
Averaging Window

\item {} 
\sphinxAtStartPar
Maximum Data Burst

\end{itemize}

\sphinxAtStartPar
Note that while the preceding discussion could be interpreted to imply a
one\sphinxhyphen{}to\sphinxhyphen{}one relationship between subscribers and a QCI, it is more
accurate to say that each QCI is associated with a class of traffic
(often corresponding to some type of application), where a given
subscriber might be sending and receiving traffic that belongs to
multiple classes at any given time. We explore this idea in much more
depth in a later chapter.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide13}.png}
\caption{Scheduler allocates Resource Blocks to user data streams based on
CQI feedback from receivers and the QCI parameters associated with
each class of service.}\label{\detokenize{primer:id9}}\label{\detokenize{primer:fig-scheduler}}\end{figure}


\section{2.3 New Radio (NR)}
\label{\detokenize{primer:new-radio-nr}}
\sphinxAtStartPar
We conclude by noting that while the previous section describes 5G as
introducing additional degrees of freedom into how data is scheduled
for transmission, the end result is a qualitatively more powerful
radio. This new 5G air interface specification, which is commonly
referred to as \sphinxstyleemphasis{New Radio (NR)}, enables three new use cases that go
well beyond simply delivering increased bandwidth:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Extreme Mobile Broadband

\item {} 
\sphinxAtStartPar
Ultra\sphinxhyphen{}Reliable Low\sphinxhyphen{}Latency Communications

\item {} 
\sphinxAtStartPar
Massive Machine\sphinxhyphen{}Type Communications

\end{itemize}

\sphinxAtStartPar
All three correspond to the requirements introduced in Chapter 1, and
can be attributed to four fundamental improvements in how 5G
multiplexes data onto the radio spectrum.

\sphinxAtStartPar
The first is the one identified in the previous section: being able to
change the waveform. This effectively introduces the ability to
dynamically change the size and number of schedulable resource units,
which opens the door to making fine\sphinxhyphen{}grain scheduling decisions that
are critical to predictable, low\sphinxhyphen{}latency communication.

\sphinxAtStartPar
The second is related to the “Multiple Access” aspect of how distinct
traffic sources are multiplexed onto the available spectrum. In 4G,
multiplexing happens in both the frequency and time domains for
downstream traffic (as described in Section 2.2), but multiplexing
happens in only the frequency domain for upstream traffic. 5G NR
multiplexes both upstream and downstream traffic in both the time and
frequency domains. Doing so provides finer\sphinxhyphen{}grain scheduling control
needed by latency\sphinxhyphen{}sensitive applications.

\sphinxAtStartPar
The third is related to the new spectrum available to 5G NR, with the
mmWave allocations opening above 24 GHz being especially
important. This is not only because of the abundance of capacity—which
makes it possible to set aside dedicated capacity for mission\sphinxhyphen{}critical
applications that require low\sphinxhyphen{}latency communication—but also because
the higher\sphinxhyphen{}frequency enables even finer\sphinxhyphen{}grain resource blocks (e.g.,
scheduling intervals as short as 0.125 ms). Again, this improves
scheduling granularity to the benefit of applications that cannot
tolerate unpredictable latency.

\sphinxAtStartPar
The fourth is related to delivering mobile connectivity to a massive
number of IoT devices, ranging from devices that require mobility
support and modest data rates (e.g. wearables, asset trackers) to
devices that support intermittent transmission of a few bytes of data
(e.g., sensors, meters). None of these devices are particularly
latency\sphinxhyphen{}sensitive or bandwidth\sphinxhyphen{}hungry, but the latter are especially
challenging because they require long battery lifetimes, and hence,
reduced hardware complexity that draws less power.

\sphinxAtStartPar
Support for IoT device connectivity revolves around allocating some of
the available radio spectrum to a light\sphinxhyphen{}weight (simplified) air
interface.  This approach started with Release 13 of LTE via two
complementary technologies: mMTC and NB\sphinxhyphen{}IoT (NarrowBand\sphinxhyphen{}IoT).  Both
technologies build on a significantly simplified version of LTE—i.e.,
limiting the numerology and flexibility needed achieve high spectrum
utilization—so as to allow for simpler IoT hardware design. mMTC
delivers up to 1 Mbps over a 1.4 MHz of bandwidth and NB\sphinxhyphen{}IoT delivers a
few tens of kbps over 200 kHz of bandwidth; hence the term
\sphinxstyleemphasis{NarrowBand}.  Both technologies have been designed to support over 1
million devices per square kilometer. With Release 16, both
technologies can be operated in\sphinxhyphen{}band with 5G, but still based on LTE
numerology. Starting with Release 17, a simpler version of 5G NR,
called \sphinxstyleemphasis{NR\sphinxhyphen{}Light}, will be introduced as the evolution of mMTC.
NR\sphinxhyphen{}Light is expected to scale the device density even further.

\sphinxAtStartPar
As a consequence of all four improvements, 5G NR is designed to
support partitioning the available bandwidth, with different
partitions dynamically allocated to different classes of traffic
(e.g., high\sphinxhyphen{}bandwidth, low\sphinxhyphen{}latency, and low\sphinxhyphen{}complexity). This is the
essence of \sphinxstyleemphasis{slicing}, an idea we will revisit throughout this book.
Moreover, once traffic with different requirements can be served by
different slices, 5G NR’s approach to multiplexing is general enough
to support varied scheduling decisions for those slices, each tailored
for the target traffic.


\chapter{Chapter 3:  Basic Architecture}
\label{\detokenize{arch:chapter-3-basic-architecture}}\label{\detokenize{arch::doc}}
\sphinxAtStartPar
This chapter identifies the main architectural components of cellular
access networks. It focuses on the components that are common to both 4G
and 5G and, as such, establishes a foundation for understanding the
advanced features of 5G presented in later chapters.

\sphinxAtStartPar
This overview is partly an exercise in introducing 3GPP terminology. For
someone that is familiar with the Internet, this terminology can seem
arbitrary (e.g., “eNB” is a “base station”), but it is important to keep
in mind that this terminology came out of the 3GPP standardization
process, which has historically been concerned about telephony and
almost completely disconnected from the IETF and other Internet\sphinxhyphen{}related
efforts. To further confuse matters, the 3GPP terminology often changes
with each generation (e.g., a base station is called eNB in 4G and gNB
in 5G). We address situations like this by using generic terminology
(e.g., base station), and referencing the 3GPP\sphinxhyphen{}specific counterpart only
when the distinction is helpful.

\phantomsection\label{\detokenize{arch:reading-terminology}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
This example is only the tip of the terminology iceberg. For a
slightly broader perspective on the complexity of terminology in
5G, see Marcin Dryjanski’s blog post: \sphinxhref{https://www.grandmetric.com/blog/2018/07/14/lte-and-5g-differences-system-complexity/}{LTE and 5G Differences:
System Complexity}.
July 2018.
\end{sphinxadmonition}


\section{3.1 Main Components}
\label{\detokenize{arch:main-components}}
\sphinxAtStartPar
The cellular network provides wireless connectivity to devices that are
on the move. These devices, which are known as \sphinxstyleemphasis{User Equipment (UE)},
have traditionally corresponded to smartphones and tablets, but will
increasingly include cars, drones, industrial and agricultural machines,
robots, home appliances, medical devices, and so on.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide01}.png}
\caption{Cellular networks consists of a Radio Access Network
(RAN) and a Mobile Core.}\label{\detokenize{arch:id2}}\label{\detokenize{arch:fig-cellular}}\end{figure}

\sphinxAtStartPar
As shown in \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}, the cellular network
consists of two main subsystems: the \sphinxstyleemphasis{Radio Access Network (RAN)} and
the \sphinxstyleemphasis{Mobile Core}. The RAN manages the radio spectrum, making sure it
is used efficiently and meets the quality\sphinxhyphen{}of\sphinxhyphen{}service requirements of
every user.  It corresponds to a distributed collection of base
stations. As noted above, in 4G these are (somewhat cryptically) named
\sphinxstyleemphasis{eNodeB} (or \sphinxstyleemphasis{eNB}), which is short for \sphinxstyleemphasis{evolved Node B}.  In 5G they
are known as \sphinxstyleemphasis{gNB}. (The g stands for “next Generation”.)

\sphinxAtStartPar
The Mobile Core is a bundle of functionality (as opposed to a
device) that serves several purposes.
\begin{itemize}
\item {} 
\sphinxAtStartPar
Provides Internet (IP) connectivity for both data and voice services.

\item {} 
\sphinxAtStartPar
Ensures this connectivity fulfills the promised QoS requirements.

\item {} 
\sphinxAtStartPar
Tracks user mobility to ensure uninterrupted service.

\item {} 
\sphinxAtStartPar
Tracks subscriber usage for billing and charging.

\end{itemize}

\sphinxAtStartPar
Note that Mobile Core is another example of a generic term. In 4G
this is called the \sphinxstyleemphasis{Evolved Packet Core (EPC)} and in 5G it is called
the \sphinxstyleemphasis{Next Generation Core (NG\sphinxhyphen{}Core)}.

\sphinxAtStartPar
Even though the word “Core” is in its name, from an Internet
perspective, the Mobile Core is still part of the access network,
effectively providing a bridge between the RAN in some geographic area
and the greater IP\sphinxhyphen{}based Internet. 3GPP provides significant
flexibility in how the Mobile Core is geographically deployed, but for
our purposes, assuming each instantiation of the Mobile Core serves a
metropolitan area is a good working model. The corresponding RAN would
then span several dozens (or even hundreds) of cell towers.

\sphinxAtStartPar
Taking a closer look at \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}}, we see that a
\sphinxstyleemphasis{Backhaul Network} interconnects the base stations that implement the RAN with
the Mobile Core. This network is typically wired, may or may not have
the ring topology shown in the Figure, and is often constructed from
commodity components found elsewhere in the Internet. For example, the
\sphinxstyleemphasis{Passive Optical Network (PON)} that implements Fiber\sphinxhyphen{}to\sphinxhyphen{}the\sphinxhyphen{}Home is a
prime candidate for implementing the RAN backhaul. The backhaul network
is obviously a necessary part of the RAN, but it is an implementation
choice and not prescribed by the 3GPP standard.

\sphinxAtStartPar
Although 3GPP specifies all the elements that implement the RAN and
Mobile Core in an open standard—including sub\sphinxhyphen{}layers we have not yet
introduced—network operators have historically bought proprietary
implementations of each subsystem from a single vendor. This lack of an
open source implementation contributes to the perceived “opaqueness” of
the cellular network in general, and the RAN in particular. And while it
is true that an eNodeB implementation does contain sophisticated
algorithms for scheduling transmission on the radio spectrum—algorithms
that are considered valuable intellectual property of the equipment
vendors—there is significant opportunity to open and disaggregate both
the RAN and the Mobile Core. The following two sections describe each,
in turn.

\sphinxAtStartPar
Before getting to those details, \hyperref[\detokenize{arch:fig-cups}]{Figure \ref{\detokenize{arch:fig-cups}}}
redraws components from \hyperref[\detokenize{arch:fig-cellular}]{Figure \ref{\detokenize{arch:fig-cellular}}} to
highlight two important distinctions. The first is that a base station has an analog
component (depicted by an antenna) and a digital component (depicted
by a processor pair). The second is that the Mobile Core is
partitioned into a \sphinxstyleemphasis{Control Plane} and \sphinxstyleemphasis{User Plane}, which is similar
to the control/data plane split that someone familiar with the
Internet would recognize. (3GPP also recently introduced a
corresponding acronym—\sphinxstyleemphasis{CUPS, Control and User Plane Separation}—to
denote this idea.) The importance of these two distinctions will
become clear in the following discussion.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide02}.png}
\caption{Mobile Core divided into a Control Plan and a User
Plane, an architectural feature known as CUPS: Control and User
Plane Separation}\label{\detokenize{arch:id3}}\label{\detokenize{arch:fig-cups}}\end{figure}


\section{3.2 Radio Access Network}
\label{\detokenize{arch:radio-access-network}}
\sphinxAtStartPar
We now describe the RAN by sketching the role each base station plays.
Keep in mind this is kind of like describing the Internet by explaining
how a router works—a not unreasonable place to start, but it doesn’t
fully do justice to the end\sphinxhyphen{}to\sphinxhyphen{}end story.

\sphinxAtStartPar
First, each base station establishes the wireless channel for a
subscriber’s UE upon power\sphinxhyphen{}up or upon handover when the UE is active.
This channel is released when the UE remains idle for a predetermined
period of time. Using 3GPP terminology, this wireless channel is said to
provide a \sphinxstyleemphasis{bearer service}. The term “bearer” has historically been used in
telecommunications (including early wireline technologies like
ISDN) to denote a data channel, as opposed to a channel that
carries signaling information.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide03}.png}
\caption{Base Station detects (and connects to) active UEs.}\label{\detokenize{arch:id4}}\label{\detokenize{arch:fig-active-ue}}\end{figure}

\sphinxAtStartPar
Second, each base station establishes “3GPP Control Plane”
connectivity between the UE and the corresponding Mobile Core Control
Plane component, and forwards signaling traffic between the two. This
signaling traffic enables UE authentication, registration, and
mobility tracking.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide04}.png}
\caption{Base Station establishes control plane connectivity
between each UE and the Mobile Core.}\label{\detokenize{arch:id5}}\label{\detokenize{arch:fig-control-plane}}\end{figure}

\sphinxAtStartPar
Third, for each active UE, the base station establishes one or more
tunnels between the corresponding Mobile Core User Plane component.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide05}.png}
\caption{Base station establishes one or more tunnels between
each UE and the Mobile Core’s User Plane.}\label{\detokenize{arch:id6}}\label{\detokenize{arch:fig-user-plane}}\end{figure}

\sphinxAtStartPar
Fourth, the base station forwards both control and user plane packets
between the Mobile Core and the UE. These packets are tunnelled over
SCTP/IP and GTP/UDP/IP, respectively. SCTP (Stream Control Transport
Protocol) is an alternative reliable transport to TCP, tailored to carry
signaling (control) information for telephony services. GTP (a nested
acronym corresponding to (General Packet Radio Service) Tunneling
Protocol) is a 3GPP\sphinxhyphen{}specific tunneling protocol designed to run over
UDP.

\sphinxAtStartPar
As an aside, it is noteworthy that connectivity between the RAN and the
Mobile Core is IP\sphinxhyphen{}based. This was introduced as one of the main changes
between 3G and 4G. Prior to 4G, the internals of the cellular network
were circuit\sphinxhyphen{}based, which is not surprising given its origins as a voice
network.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide06}.png}
\caption{Base Station to Mobile Core (and Base Station to Base
Station) control plane tunneled over SCTP/IP and user plane
tunneled over GTP/UDP/IP.}\label{\detokenize{arch:id7}}\label{\detokenize{arch:fig-tunnels}}\end{figure}

\sphinxAtStartPar
Fifth, each base station coordinates UE handovers with neighboring
base stations, using direct station\sphinxhyphen{}to\sphinxhyphen{}station links. Exactly like the
station\sphinxhyphen{}to\sphinxhyphen{}core connectivity shown in the previous figure, these links
are used to transfer both control plane (SCTP over IP) and user plane
(GTP over UDP/IP) packets.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide07}.png}
\caption{Base Stations cooperate to implement UE hand over.}\label{\detokenize{arch:id8}}\label{\detokenize{arch:fig-handover}}\end{figure}

\sphinxAtStartPar
Sixth, the base stations coordinate wireless multi\sphinxhyphen{}point transmission to
a UE from multiple base stations, which may or may not be part of a UE
handover from one base station to another.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide08}.png}
\caption{Base Stations cooperate to implement multipath
transmission (link aggregation) to UEs.}\label{\detokenize{arch:id9}}\label{\detokenize{arch:fig-link-aggregation}}\end{figure}

\sphinxAtStartPar
The main takeaway is that the base station can be viewed as a
specialized forwarder. In the Internet\sphinxhyphen{}to\sphinxhyphen{}UE direction, it fragments
outgoing IP packets into physical layer segments and schedules them
for transmission over the available radio spectrum, and in the
UE\sphinxhyphen{}to\sphinxhyphen{}Internet direction it assembles physical layer segments into IP
packets and forwards them (over a GTP/UDP/IP tunnel) to the upstream
user plane of the Mobile Core. Also, based on observations of the
wireless channel quality and per\sphinxhyphen{}subscriber policies, it decides
whether to (a) forward outgoing packets directly to the UE, (b)
indirectly forward packets to the UE via a neighboring base station,
or (c) utilize multiple paths to reach the UE. The third case has the
option of either spreading the physical payloads across multiple base
stations or across multiple carrier frequencies of a single base
station (including Wi\sphinxhyphen{}Fi).

\sphinxAtStartPar
Note that as outlined in Chapter 2, scheduling is complex and
multi\sphinxhyphen{}faceted, even when viewed as a localized decision at a single
base station. What we now see is that there is also a global element,
whereby it’s possible to forward traffic to a different base station
(or to multiple base stations) in an effort to make efficient use of
the radio spectrum over a larger geographic area.

\sphinxAtStartPar
In other words, the RAN as a whole (i.e., not just a single base
station) not only supports handovers (an obvious requirement for
mobility), but also \sphinxstyleemphasis{link aggregation} and \sphinxstyleemphasis{load balancing}, mechanisms
that are familiar to anyone who understands the Internet. We will
revisit how such RAN\sphinxhyphen{}wide (global) decisions can be made using SDN
techniques in a later chapter.


\section{3.3 Mobile Core}
\label{\detokenize{arch:mobile-core}}
\sphinxAtStartPar
The main function of the Mobile Core is to provide external packet data
network (i.e., Internet) connectivity to mobile subscribers, while
ensuring that they are authenticated and their observed service
qualities satisfy their subscription SLAs. An important aspect of the
Mobile Core is that it needs to manage all subscribers’ mobility by
keeping track of their last whereabouts at the granularity of the
serving base station. It’s the fact that the Mobile Core is keeping
track of individual subscribers—something that the Internet’s core
does not do—that creates a lot of the complexity in its architecture,
especially given that those subscribers are moving around.

\sphinxAtStartPar
While the aggregate functionality remains largely the same as we migrate
from 4G to 5G, how that functionality is virtualized and factored into
individual components changes. The 5G Mobile Core is heavily
influenced by the cloud’s march toward a microservice\sphinxhyphen{}based (cloud
native) architecture. This shift to cloud native is deeper than it might
first appear, in part because it opens the door to customization and
specialization. Instead of supporting just voice and broadband
connectivity, the 5G Mobile Core can evolve to also support, for
example, massive IoT, which has a fundamentally different latency
requirement and usage pattern (i.e., many more devices connecting
intermittently). This stresses—if not breaks—a one\sphinxhyphen{}size\sphinxhyphen{}fits\sphinxhyphen{}all
approach to session management.


\subsection{4G Mobile Core}
\label{\detokenize{arch:g-mobile-core}}
\sphinxAtStartPar
The 4G Mobile Core, which 3GPP officially refers to as the \sphinxstyleemphasis{Evolved
Packet Core (EPC)}, consists of five main components, the first three of
which run in the Control Plane (CP) and the second two of which run in
the User Plane (UP).
\begin{itemize}
\item {} 
\sphinxAtStartPar
MME (Mobility Management Entity): Tracks and manages the movement of
UEs throughout the RAN. This includes recording when the UE is not
active.

\item {} 
\sphinxAtStartPar
HSS (Home Subscriber Server): A database that contains all
subscriber\sphinxhyphen{}related information.

\item {} 
\sphinxAtStartPar
PCRF (Policy \& Charging Rules Function): Tracks and manages policy
rules and records billing data on subscriber traffic.

\item {} 
\sphinxAtStartPar
SGW (Serving Gateway): Forwards IP packets to and from the RAN.
Anchors the Mobile Core end of the bearer service to a (potentially
mobile) UE, and so is involved in handovers from one base station to
another.

\item {} 
\sphinxAtStartPar
PGW (Packet Gateway): Essentially an IP router, connecting the Mobile
Core to the external Internet. Supports additional access\sphinxhyphen{}related
functions, including policy enforcement, traffic shaping, and
charging.

\end{itemize}

\sphinxAtStartPar
Although specified as distinct components, in practice the SGW
(RAN\sphinxhyphen{}facing) and PGW (Internet\sphinxhyphen{}facing) are often combined in a single
device, commonly referred to as an S/PGW. The end result is illustrated
in \hyperref[\detokenize{arch:fig-4g-core}]{Figure \ref{\detokenize{arch:fig-4g-core}}}.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide20}.png}
\caption{4G Mobile Core (Evolved Packet Core).}\label{\detokenize{arch:id10}}\label{\detokenize{arch:fig-4g-core}}\end{figure}

\sphinxAtStartPar
Note that 3GPP is flexible in how the Mobile Core components are
deployed to serve a geographic area. For example, a single MME/PGW pair
might serve a metropolitan area, with SGWs deployed across \textasciitilde{}10 edge
sites spread throughout the city, each of which serves \textasciitilde{}100 base
stations. But alternative deployment configurations are allowed by the
spec.


\subsection{5G Mobile Core}
\label{\detokenize{arch:id1}}
\sphinxAtStartPar
The 5G Mobile Core, which 3GPP calls the \sphinxstyleemphasis{NG\sphinxhyphen{}Core}, adopts a
microservice\sphinxhyphen{}like architecture, where we say “microservice\sphinxhyphen{}like” because
while the 3GPP specification spells out this level of disaggregation, it
is really just prescribing a set of functional blocks and not an
implementation. A set of functional blocks is very
different from the collection of engineering decisions that go into
designing a microservice\sphinxhyphen{}based system. That said, viewing the collection of
components shown in \hyperref[\detokenize{arch:fig-5g-core}]{Figure \ref{\detokenize{arch:fig-5g-core}}}
as a set of microservices is a good working model.

\sphinxAtStartPar
The following organizes the set of functional blocks into three groups.
The first group runs in the Control Plane (CP) and has a counterpart in
the EPC.
\begin{itemize}
\item {} 
\sphinxAtStartPar
AMF (Core Access and Mobility Management Function): Responsible for connection
and reachability management, mobility management, access
authentication and authorization, and location services. Manages the
mobility\sphinxhyphen{}related aspects of the EPC’s MME.

\item {} 
\sphinxAtStartPar
SMF (Session Management Function): Manages each UE session, including
IP address allocation, selection of associated UP function, control
aspects of QoS, and control aspects of UP routing. Roughly
corresponds to part of the EPC’s MME and the control\sphinxhyphen{}related aspects
of the EPC’s PGW.

\item {} 
\sphinxAtStartPar
PCF (Policy Control Function): Manages the policy rules that other CP
functions then enforce. Roughly corresponds to the EPC’s PCRF.

\item {} 
\sphinxAtStartPar
UDM (Unified Data Management): Manages user identity, including the
generation of authentication credentials. Includes part of the
functionality in the EPC’s HSS.

\item {} 
\sphinxAtStartPar
AUSF (Authentication Server Function): Essentially an authentication
server. Includes part of the functionality in the EPC’s HSS.

\end{itemize}

\sphinxAtStartPar
The second group also runs in the Control Plane (CP) but does not have
a direct counterpart in the EPC:
\begin{itemize}
\item {} 
\sphinxAtStartPar
SDSF (Structured Data Storage Network Function): A “helper” service
used to store structured data. Could be implemented by an “SQL
Database” in a microservices\sphinxhyphen{}based system.

\item {} 
\sphinxAtStartPar
UDSF (Unstructured Data Storage Network Function): A “helper” service
used to store unstructured data. Could be implemented by a “Key/Value
Store” in a microservices\sphinxhyphen{}based system.

\item {} 
\sphinxAtStartPar
NEF (Network Exposure Function): A means to expose select
capabilities to third\sphinxhyphen{}party services, including translation between
internal and external representations for data. Could be implemented
by an “API Server” in a microservices\sphinxhyphen{}based system.

\item {} 
\sphinxAtStartPar
NRF (NF Repository Function): A means to discover available services.
Could be implemented by a “Discovery Service” in a
microservices\sphinxhyphen{}based system.

\item {} 
\sphinxAtStartPar
NSSF (Network Slicing Selector Function): A means to select a Network
Slice to serve a given UE. Network slices are essentially a way to
partition network resources in order to
differentiate service given to different users. It is a key feature
of 5G that we discuss in depth in a later chapter.

\end{itemize}

\sphinxAtStartPar
The third group includes the one component that runs in the User Plane
(UP):
\begin{itemize}
\item {} 
\sphinxAtStartPar
UPF (User Plane Function): Forwards traffic between RAN and the
Internet, corresponding to the S/PGW combination in EPC. In addition
to packet forwarding, it is responsible for policy enforcement, lawful
intercept, traffic usage reporting, and QoS policing.

\end{itemize}

\sphinxAtStartPar
Of these, the first and third groups are best viewed as a
straightforward refactoring of 4G’s EPC, while the second group—despite
the gratuitous introduction of new terminology—is 3GPP’s way of pointing
to a cloud native solution as the desired end\sphinxhyphen{}state for the Mobile Core.
Of particular note, introducing distinct storage services means that all
the other services can be stateless, and hence, more readily scalable.
Also note that \hyperref[\detokenize{arch:fig-5g-core}]{Figure \ref{\detokenize{arch:fig-5g-core}}} adopts an idea that’s
common in microservice\sphinxhyphen{}based systems, namely, to show a \sphinxstyleemphasis{message bus}
interconnecting all the components rather than including a full set of
pairwise connections. This also suggests a well\sphinxhyphen{}understood
implementation strategy.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide33}.png}
\caption{5G Mobile Core (NG\sphinxhyphen{}Core).}\label{\detokenize{arch:id11}}\label{\detokenize{arch:fig-5g-core}}\end{figure}

\sphinxAtStartPar
Stepping back from these details, and with the caveat that we are
presuming an implementation, the main takeaway is that we can
conceptualize the Mobile Core as a graph of services. You will
sometimes hear this called a \sphinxstyleemphasis{Service Graph} or \sphinxstyleemphasis{Service Chain}, the latter being more prevalent
in NFV\sphinxhyphen{}oriented documents. Another term, \sphinxstyleemphasis{Service Mesh}, has taken on
a rather specific meaning in cloud native terminology—we’ll avoid
overloading that term here. 3GPP is silent on the specific terminology
since it is considered an implementation choice rather than part of the
specification. We describe our implementation choices in later chapters.


\section{3.4 Security and Mobility}
\label{\detokenize{arch:security-and-mobility}}
\sphinxAtStartPar
We now take a closer look at two unique features of the cellular
network—its support for security and mobility—both of which
differentiate it from WiFi. The following also serves to fill in some
details about how each individual UE connects to the network.

\sphinxAtStartPar
We start with the security architecture, which is grounded in two
trust assumptions.  First, each Base Station trusts that it is
connected to the Mobile Core by a secure private network, over which
it establishes the tunnels introduced in \hyperref[\detokenize{arch:fig-tunnels}]{Figure \ref{\detokenize{arch:fig-tunnels}}}: a GTP/UDP/IP tunnel to the Core’s User Plane (Core\sphinxhyphen{}UP)
and a SCTP/IP tunnel to the Core’s Control Plane (Core\sphinxhyphen{}CP). Second,
each UE has an operator\sphinxhyphen{}provided SIM card, which uniquely identifies
the subscriber (i.e., phone number) and establishes the radio
parameters (e.g., frequency band) needed to communicate with that
operator’s Base Stations. The SIM card also includes a secret key that
the UE uses to authenticate itself.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide34}.png}
\caption{Sequence of steps to establish secure Control and User Plane
channels.}\label{\detokenize{arch:id12}}\label{\detokenize{arch:fig-secure}}\end{figure}

\sphinxAtStartPar
With this starting point, \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} shows the
per\sphinxhyphen{}UE connection sequence. When a UE first becomes active, it
communicates with a nearby Base Station over a temporary
(unauthenticated) radio link (Step 1).  The Base Station forwards the
request to the Core\sphinxhyphen{}CP over the existing tunnel, and the Core\sphinxhyphen{}CP
(specifically, the MME in 4G and the AMF in 5G) initiates an
authentication protocol with the UE (Step 2). 3GPP identifies a set of
options for authentication and encryption, where the actual protocols
used are an implementation choice. For example, \sphinxstyleemphasis{Advanced Encryption
Standard} (AES) is one of the options for encryption. Note that this
authentication exchange is initially in the clear since the Base
Station to UE link is not yet secure.

\sphinxAtStartPar
Once the UE and Core\sphinxhyphen{}CP are satisfied with each other’s identity, the
Core\sphinxhyphen{}CP informs the other components of the parameters they will need
to service the UE (Step 3). This includes: (a) instructing the Core\sphinxhyphen{}UP
to initialize the user plane (e.g., assign an IP address to the UE and
set the appropriate QCI parameter); (b) instructing the Base Station
to establish an encrypted channel to the UE; and (c) giving the UE the
symmetric key it will need to use the encrypted channel with the Base
Station.  The symmetric key is encrypted using the public key of the
UE (so only the UE can decrypt it, using its secret key). Once
complete, the UE can use the end\sphinxhyphen{}to\sphinxhyphen{}end user plane channel through the
Core\sphinxhyphen{}UP (Step 4).

\sphinxAtStartPar
There are three additional details of note about this process. First,
the secure control channel between the UE and the Core\sphinxhyphen{}CP set up
during Step 2 remains available, and is used by the Core\sphinxhyphen{}CP to send
additional control instructions to the UE during the course of the
session.

\sphinxAtStartPar
Second, the user plane channel established during Step 4 is referred
to as the \sphinxstyleemphasis{Default Bearer Service}, but additional channels can be
established between the UE and Core\sphinxhyphen{}UP, each with a potentially
different QCI value. This might be done on an
application\sphinxhyphen{}by\sphinxhyphen{}application basis, for example, under the control of
the Mobile Core doing \sphinxstyleemphasis{Deep Packet Inspection} (DPI) on the traffic,
looking for flows that require special treatment.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide35}.png}
\caption{Sequence of per\sphinxhyphen{}hop tunnels involved in an end\sphinxhyphen{}to\sphinxhyphen{}end User Plane
channel.}\label{\detokenize{arch:id13}}\label{\detokenize{arch:fig-per-hop}}\end{figure}

\sphinxAtStartPar
Third, while the resulting user plane channels are logically
end\sphinxhyphen{}to\sphinxhyphen{}end, each is actually implemented as a sequence of per\sphinxhyphen{}hop
tunnels, as illustrated in \hyperref[\detokenize{arch:fig-per-hop}]{Figure \ref{\detokenize{arch:fig-per-hop}}}.  (The
figure shows the SGW and PGW from the 4G Mobile Core to make the
example more concrete.) This means each component on the end\sphinxhyphen{}to\sphinxhyphen{}end
path terminates a downstream tunnel using one local identifier for a
given UE, and initiates an upstream tunnel using a second local
identifier for that UE. In practice, these per\sphinxhyphen{}flow tunnels are often
bundled into an single inter\sphinxhyphen{}component tunnel, which makes it
impossible to differentiate the level of service given to any
particular end\sphinxhyphen{}to\sphinxhyphen{}end UE channel. This is a limitation of 4G that 5G
has ambitions to correct.

\sphinxAtStartPar
Support for mobility can now be understood as the process of
re\sphinxhyphen{}executing one or more of the steps shown in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}} as the UE moves throughout the RAN.  The unauthenticated
link indicated by (1) allows the UE to be known to all Base Station
within range. (We refer to these as \sphinxstyleemphasis{potential links} in later
chapters.) Based on the signal’s measured CQI, the Base Stations
communicate directly with each other to make a handover decision. Once
made, the decision is then communicated to the Mobile Core,
re\sphinxhyphen{}triggering the setup functions indicated by (3), which in turn
re\sphinxhyphen{}builds the user plane tunnel between the Base Station and the SGW
shown in \hyperref[\detokenize{arch:fig-per-hop}]{Figure \ref{\detokenize{arch:fig-per-hop}}} (or correspondingly,
between the Base Station and the UPF in 5G). One of the most unique
features of the cellular network is that the Mobile Core’s user plane
(e.g., UPF in 5G) buffers data during the handover transition,
avoiding dropped packets and subsequent end\sphinxhyphen{}to\sphinxhyphen{}end retransmissions.

\sphinxAtStartPar
In other words, the cellular network maintains the \sphinxstyleemphasis{UE session} in the
face of mobility (corresponding to the control and data channels
depicted by (2) and (4) in \hyperref[\detokenize{arch:fig-secure}]{Figure \ref{\detokenize{arch:fig-secure}}},
respectively), but it is able to do so only when the same Mobile Core
serves the UE (i.e., only the Base Station changes).  This would
typically be the case for a UE moving within a metropolitan area.
Moving between metro areas—and hence, between Mobile Cores—is
indistinguishable from power cycling a UE. The UE is assigned a new IP
address and no attempt is made to buffer and subsequently deliver
in\sphinxhyphen{}flight data. Independent of mobility, but relevant to this
discussion, any UE that becomes inactive for a period of time also
loses its session, with a new session established and a new IP address
assigned when the UE becomes active again.

\sphinxAtStartPar
Note that this session\sphinxhyphen{}based approach can be traced to the cellular
network’s roots as a connection\sphinxhyphen{}oriented network. An interesting
thought experiment is whether the Mobile Core will continue to evolve
so as to better match the connectionless assumptions of the Internet
protocols that typically run on top of it.


\section{3.5 Deployment Options}
\label{\detokenize{arch:deployment-options}}
\sphinxAtStartPar
With an already deployed 4G RAN/EPC in the field and a new 5G
RAN/NG\sphinxhyphen{}Core deployment underway, we can’t ignore the issue of
transitioning from 4G to 5G (an issue the IP\sphinxhyphen{}world has been grappling
with for 20 years). 3GPP officially spells out multiple deployment
options, which can be summarized as follows.
\begin{itemize}
\item {} 
\sphinxAtStartPar
Standalone 4G / Stand\sphinxhyphen{}Alone 5G

\item {} 
\sphinxAtStartPar
Non\sphinxhyphen{}Standalone (4G+5G RAN) over 4G’s EPC

\item {} 
\sphinxAtStartPar
Non\sphinxhyphen{}Standalone (4G+5G RAN) over 5G’s NG\sphinxhyphen{}Core

\end{itemize}

\sphinxAtStartPar
The second of the three options, which is generally referred to as
“NSA“, involves 5G base stations being deployed alongside the
existing 4G base stations in a given geography to provide a data\sphinxhyphen{}rate
and capacity boost. In NSA, control plane traffic between the user
equipment and the 4G Mobile Core utilizes (i.e., is forwarded through)
4G base stations, and the 5G base stations are used only to carry user
traffic. Eventually, it is expected that operators complete their
migration to 5G by deploying NG Core and connecting their 5G base
stations to it for Standalone (SA) operation. NSA and SA operations
are illustrated in \hyperref[\detokenize{arch:fig-nsa}]{Figure \ref{\detokenize{arch:fig-nsa}}}.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide38}.png}
\caption{NSA and SA options for 5G deployment.}\label{\detokenize{arch:id14}}\label{\detokenize{arch:fig-nsa}}\end{figure}

\sphinxAtStartPar
One reason we call attention to the phasing issue is that we face a
similar challenge in the chapters that follow. The closer the following
discussion gets to implementation details, the more specific we have to
be about whether we are using 4G components or 5G components. As a
general rule, we use 4G components—particularly with respect to the
Mobile Core, since that’s what’s available in open source today—and trust
the reader can make the appropriate substitution without loss of
generality. Like the broader industry, the open source community is in
the process of incrementally evolving its 4G code base into its
5G\sphinxhyphen{}compliant counterpart.

\phantomsection\label{\detokenize{arch:reading-migration}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For more insight into 4G to 5G migration strategies, see
\sphinxhref{https://www.gsma.com/futurenetworks/wp-content/uploads/2018/04/Road-to-5G-Introduction-and-Migration\_FINAL.pdf}{Road to 5G: Introduction and Migration}.
GSMA Report, April 2018.
\end{sphinxadmonition}


\chapter{Chapter 4:  RAN Internals}
\label{\detokenize{ran:chapter-4-ran-internals}}\label{\detokenize{ran::doc}}
\sphinxAtStartPar
The description of the RAN in the previous chapter focused on
functionality, but was mostly silent about the RAN’s internal
structure. We now focus on some of the internal details, and in
doing so, explain how the RAN is being transformed in 5G. This
involves first describing the stages in the packet processing
pipeline, and then showing how these stages can be disaggregated,
distributed and implemented.

\sphinxAtStartPar
Our approach in this chapter is to incrementally build the RAN from
the bottom up in the first three sections. Section 4.4 then summarizes
the overall design, with a focus on how the resulting end\sphinxhyphen{}to\sphinxhyphen{}end
system is architected to evolve.


\section{4.1 Packet Processing Pipeline}
\label{\detokenize{ran:packet-processing-pipeline}}
\sphinxAtStartPar
\hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} shows the packet processing stages
implemented by the base station. These stages are specified by the 3GPP
standard. Note that the figure depicts the base station as a pipeline
(running left\sphinxhyphen{}to\sphinxhyphen{}right for packets sent to the UE) but it is equally valid to view it as a protocol
stack (as is typically done in official 3GPP documents). Also note that
(for now) we are agnostic as to how these stages are implemented, but
since we are ultimately heading towards a cloud\sphinxhyphen{}based implementation,
you can think of each as corresponding to a microservice (if that is
helpful).

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide14}.png}
\caption{RAN processing pipeline, including both user and
control plane components.}\label{\detokenize{ran:id1}}\label{\detokenize{ran:fig-pipeline}}\end{figure}

\sphinxAtStartPar
The key stages are as follows.
\begin{itemize}
\item {} 
\sphinxAtStartPar
RRC (Radio Resource Control) \(\rightarrow\) Responsible for configuring the
coarse\sphinxhyphen{}grain and policy\sphinxhyphen{}related aspects of the pipeline. The RRC runs
in the RAN’s control plane; it does not process packets on the user
plane.

\item {} 
\sphinxAtStartPar
PDCP (Packet Data Convergence Protocol) \(\rightarrow\) Responsible for compressing
and decompressing IP headers, ciphering and integrity protection, and
making an “early” forwarding decision (i.e., whether to send the
packet down the pipeline to the UE or forward it to another base
station).

\item {} 
\sphinxAtStartPar
RLC (Radio Link Control) \(\rightarrow\) Responsible for segmentation and
reassembly, including reliably transmitting/receiving segments by
implementing a form of ARQ (automatic repeat request).

\item {} 
\sphinxAtStartPar
MAC (Media Access Control) \(\rightarrow\) Responsible for buffering, multiplexing
and demultiplexing segments, including all real\sphinxhyphen{}time scheduling
decisions about what segments are transmitted when. Also able to make
a “late” forwarding decision (i.e., to alternative carrier
frequencies, including Wi\sphinxhyphen{}Fi).

\item {} 
\sphinxAtStartPar
PHY (Physical Layer) \(\rightarrow\) Responsible for coding and modulation (as
discussed in an earlier chapter), including FEC.

\end{itemize}

\sphinxAtStartPar
The last two stages in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} (D/A
conversion and the RF front\sphinxhyphen{}end) are beyond the scope of this book.

\sphinxAtStartPar
While it is simplest to view the stages in \hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}}
as a pure left\sphinxhyphen{}to\sphinxhyphen{}right pipeline, in practice the Scheduler running in the
MAC stage implements the “main loop” for outbound traffic, reading data
from the upstream RLC and scheduling transmissions to the downstream
PHY. In particular, since the Scheduler determines the number of bytes
to transmit to a given UE during each time period (based on all the
factors outlined in an earlier chapter), it must request (get) a segment
of that length from the upstream queue. In practice, the size of the
segment that can be transmitted on behalf of a single UE during a single
scheduling interval can range from a few bytes to an entire IP packet.


\section{4.2 Split RAN}
\label{\detokenize{ran:split-ran}}
\sphinxAtStartPar
The next step is to understand how the functionality outlined above is
partitioned between physical elements, and hence, “split” across
centralized and distributed locations. The dominant option has
historically been “no split,” with the entire pipeline shown in
\hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} running in the base station.  Going
forward, the 3GPP standard has been extended to allow for multiple
split\sphinxhyphen{}points, with the partition shown in \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}} being actively pursued by the operator\sphinxhyphen{}led O\sphinxhyphen{}RAN
(Open RAN) Alliance. It is the split we adopt throughout the rest of
this book.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide15}.png}
\caption{Split\sphinxhyphen{}RAN processing pipeline distributed across a
Central Unit (CU), Distributed Unit (DU), and Radio Unit (RU).}\label{\detokenize{ran:id2}}\label{\detokenize{ran:fig-split-ran}}\end{figure}

\sphinxAtStartPar
This results in a RAN\sphinxhyphen{}wide configuration similar to that shown in
\hyperref[\detokenize{ran:fig-ran-hierarchy}]{Figure \ref{\detokenize{ran:fig-ran-hierarchy}}}, where a single \sphinxstyleemphasis{Central Unit (CU)}
running in the cloud serves multiple \sphinxstyleemphasis{Distributed Units (DUs)}, each of
which in turn serves multiple \sphinxstyleemphasis{Radio Units (RUs)}. Critically, the RRC
(centralized in the CU) is responsible for only near\sphinxhyphen{}real\sphinxhyphen{}time
configuration and control decision making, while the Scheduler that is
part of the MAC stage is responsible for all real\sphinxhyphen{}time scheduling
decisions.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide16}.png}
\caption{Split\sphinxhyphen{}RAN hierarchy, with one CU serving multiple DUs,
each of which serves multiple RUs.}\label{\detokenize{ran:id3}}\label{\detokenize{ran:fig-ran-hierarchy}}\end{figure}

\sphinxAtStartPar
Because scheduling decisions for radio transmission are made by the
MAC layer in real
time, a DU needs to be “near” (within 1ms) the RUs it manages. (You
can’t afford to make scheduling decisions based on out\sphinxhyphen{}of\sphinxhyphen{}date channel
information.) One familiar configuration is
to co\sphinxhyphen{}locate a DU and an RU in a cell tower. But when an RU corresponds
to a small cell, many of which might be spread across a modestly\sphinxhyphen{}sized
geographic area (e.g., a mall, campus, or factory), then a single DU
would likely service multiple RUs. The use of mmWave in 5G is likely to
make this later configuration all the more common.

\sphinxAtStartPar
Also note that the split\sphinxhyphen{}RAN changes the nature of the Backhaul Network,
which in 4G connected the base stations (eNBs) back to the Mobile Core.
With the split\sphinxhyphen{}RAN there are multiple connections, which are officially
labeled as follows.
\begin{itemize}
\item {} 
\sphinxAtStartPar
RU\sphinxhyphen{}DU connectivity is called the Fronthaul

\item {} 
\sphinxAtStartPar
DU\sphinxhyphen{}CU connectivity is called the Midhaul

\item {} 
\sphinxAtStartPar
CU\sphinxhyphen{}Mobile Core connectivity is called the Backhaul

\end{itemize}

\sphinxAtStartPar
One observation about the CU (which is relevant in the next chapter)
is that one might co\sphinxhyphen{}locate the CU and Mobile Core in the same
cluster, meaning the backhaul is implemented in the cluster switching
fabric. In such a configuration, the midhaul then effectively serves
the same purpose as the original backhaul, and the fronthaul is
constrained by the predictable/low\sphinxhyphen{}latency requirements of the MAC
stage’s real\sphinxhyphen{}time scheduler.

\sphinxAtStartPar
A second observation about the CU shown in \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}} is that it encompasses two functional blocks—the RRC
and the PDCP—which lie on the RAN’s control plane and user plane,
respectively. This separation is consistent with the idea of CUPS
introduced in Chapter 3, and plays an increasingly important role as
we dig deeper into how the RAN is implemented. For now, we note that
the two parts are typically referred to as the CU\sphinxhyphen{}C and CU\sphinxhyphen{}U,
respectively.

\phantomsection\label{\detokenize{ran:reading-backhaul}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For more insight into design considerations for
interconnecting the distributed components of a Split RAN, see
\sphinxhref{https://www.ngmn.org/wp-content/uploads/NGMN\_RANEV\_D4\_BH\_FH\_Evolution\_V1.01.pdf}{RAN Evolution Project: Backhaul and Fronthaul Evolution}.
NGMN Alliance Report, March 2015.
\end{sphinxadmonition}


\section{4.3 Software\sphinxhyphen{}Defined RAN}
\label{\detokenize{ran:software-defined-ran}}
\sphinxAtStartPar
We now describe how the RAN is implemented according to SDN
principles, resulting in an SD\sphinxhyphen{}RAN. The key architectural insight is
shown in \hyperref[\detokenize{ran:fig-rrc-split}]{Figure \ref{\detokenize{ran:fig-rrc-split}}}, where the RRC from
\hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} is partitioned into two
sub\sphinxhyphen{}components: the one on the left provides a 3GPP\sphinxhyphen{}compliant way for
the RAN to interface to the Mobile Core’s control plane, while the one
on the right opens a new programmatic API for exerting software\sphinxhyphen{}based
control over the pipeline that implements the RAN user plane.

\sphinxAtStartPar
To be more specific, the left sub\sphinxhyphen{}component simply forwards control
packets between the Mobile Core and the PDCP, providing a path over
which the Mobile Core can communicate with the UE for control
purposes, whereas the right sub\sphinxhyphen{}component implements the core of the
RRC’s control functionality. This component is commonly referred to as
the \sphinxstyleemphasis{RAN Intelligent Controller (RIC)} in O\sphinxhyphen{}RAN architecture
documents, so we adopt this terminology.  The “Near\sphinxhyphen{}Real Time”
qualifier indicates the RIC is part of 10\sphinxhyphen{}100ms control loop implemented
in the CU, as opposed to the \textasciitilde{}1ms control loop required by the MAC
scheduler running in the DU.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide18}.png}
\caption{RRC disaggregated into a Mobile Core facing control
plane component and a Near\sphinxhyphen{}Real\sphinxhyphen{}Time Controller.}\label{\detokenize{ran:id4}}\label{\detokenize{ran:fig-rrc-split}}\end{figure}

\sphinxAtStartPar
Although not shown in \hyperref[\detokenize{ran:fig-rrc-split}]{Figure \ref{\detokenize{ran:fig-rrc-split}}}, keep in mind
(from \hyperref[\detokenize{ran:fig-split-ran}]{Figure \ref{\detokenize{ran:fig-split-ran}}}) that all constituent parts of
the RRC, plus the PDCP, form the CU.

\sphinxAtStartPar
Completing the picture, \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}} shows
the Near\sphinxhyphen{}RT RIC implemented as an SDN Controller hosting a
set of SDN control apps. The RIC maintains a \sphinxstyleemphasis{RAN Network Information
Base (R\sphinxhyphen{}NIB)}\textendash{}a common set of information that can be consumed by numerous
control apps. The R\sphinxhyphen{}NIB includes time\sphinxhyphen{}averaged CQI values and other
per\sphinxhyphen{}session state (e.g., GTP tunnel IDs, QCI values for the type of
traffic), while the MAC (as part of the DU) maintains the
instantaneous CQI values required by the real\sphinxhyphen{}time
scheduler. Specifically, the R\sphinxhyphen{}NIB includes the following state.
\begin{itemize}
\item {} 
\sphinxAtStartPar
NODES: Base Stations and Mobile Devices
\begin{itemize}
\item {} 
\sphinxAtStartPar
Base Station Attributes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identifiers

\item {} 
\sphinxAtStartPar
Version

\item {} 
\sphinxAtStartPar
Config Report

\item {} 
\sphinxAtStartPar
RRM config

\item {} 
\sphinxAtStartPar
PHY resource usage

\end{itemize}

\item {} 
\sphinxAtStartPar
Mobile Device Attributes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identifiers

\item {} 
\sphinxAtStartPar
Capability

\item {} 
\sphinxAtStartPar
Measurement Config

\item {} 
\sphinxAtStartPar
State (Active/Idle)

\end{itemize}

\end{itemize}

\item {} 
\sphinxAtStartPar
LINKS: \sphinxstyleemphasis{Actual} between two nodes and \sphinxstyleemphasis{Potential} between UEs and all
neighbor cells
\begin{itemize}
\item {} 
\sphinxAtStartPar
Link Attributes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identifiers

\item {} 
\sphinxAtStartPar
Link Type

\item {} 
\sphinxAtStartPar
Config/Bearer Parameters

\item {} 
\sphinxAtStartPar
QCI Value

\end{itemize}

\end{itemize}

\item {} 
\sphinxAtStartPar
SLICES: Virtualized RAN Construct
\begin{itemize}
\item {} 
\sphinxAtStartPar
Slice Attributes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Links

\item {} 
\sphinxAtStartPar
Bearers/Flows

\item {} 
\sphinxAtStartPar
Validity Period

\item {} 
\sphinxAtStartPar
Desired KPIs

\item {} 
\sphinxAtStartPar
MAC RRM Configuration

\item {} 
\sphinxAtStartPar
RRM Control Configuration

\end{itemize}

\end{itemize}

\end{itemize}

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=500\sphinxpxdimen]{{Slide19}.png}
\caption{Example set of control applications running on top of
Near\sphinxhyphen{}Real\sphinxhyphen{}Time RAN Controller.}\label{\detokenize{ran:id5}}\label{\detokenize{ran:fig-ran-controller}}\end{figure}

\sphinxAtStartPar
The example Control Apps in \hyperref[\detokenize{ran:fig-ran-controller}]{Figure \ref{\detokenize{ran:fig-ran-controller}}}
include a range of possibilities, but is not intended to be an
exhaustive list.  The right\sphinxhyphen{}most example, RAN Slicing, is the most
ambitious in that it introduces a new capability: Virtualizing the
RAN. It is also an idea that has been implemented, which we describe
in more detail in the next chapter.

\sphinxAtStartPar
The next three (RF Configuration, Semi\sphinxhyphen{}Persistent Scheduling, Cipher Key
Assignment) are examples of configuration\sphinxhyphen{}oriented applications. They
provide a programmatic way to manage seldom\sphinxhyphen{}changing configuration
state, thereby enabling zero\sphinxhyphen{}touch operations. Coming up with meaningful
policies (perhaps driven by analytics) is likely to be an avenue for
innovation in the future.

\sphinxAtStartPar
The left\sphinxhyphen{}most four example Control Applications are the sweet spot for
SDN, with its emphasis on central control over distributed
forwarding. These functions—Link Aggregation Control, Interference
Management, Load Balancing, and Handover Control—are currently
implemented by individual base stations with only local visibility,
but they have global consequences. The SDN approach is to collect the
available input data centrally, make a globally optimal decision, and
then push the respective control parameters back to the base stations
for execution. Realizing this value in the RAN is still a
work\sphinxhyphen{}in\sphinxhyphen{}progress, but products that take this approach are
emerging. Evidence using an analogous approach to optimize
wide\sphinxhyphen{}area networks over many years is compelling.

\sphinxAtStartPar
While the above loosely categorizes the space of potential control
apps as either config\sphinxhyphen{}oriented or control\sphinxhyphen{}oriented, another possible
characterization is based on the current practice of controlling the
mobile link at two different levels. At a fine\sphinxhyphen{}grain level, per\sphinxhyphen{}node
and per\sphinxhyphen{}link control is conducted using Radio Resource Management
(RRM) functions that are distributed across the individual base
stations.  RRM functions include scheduling, handover control, link
and carrier aggregation control, bearer control, and access control.
At a coarse\sphinxhyphen{}grain level, regional mobile network optimization and
configuration is conducted using \sphinxstyleemphasis{Self\sphinxhyphen{}Organizing Network (SON)}
functions. These functions oversee neighbor lists, manage load
balancing, optimize coverage and capacity, aim for network\sphinxhyphen{}wide
interference mitigation, centrally configure parameters, and so on. As
a consequence of these two levels of control, it is not uncommon to
see reference to \sphinxstyleemphasis{RRM Applications} and \sphinxstyleemphasis{SON Applications},
respectively, in O\sphinxhyphen{}RAN documents for SD\sphinxhyphen{}RAN.

\phantomsection\label{\detokenize{ran:reading-b4}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For an example of how SDN principles have been successfully applied
to a production network, we recommend \sphinxhref{https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf}{B4: Experience with a
Globally\sphinxhyphen{}Deployed Software Defined WAN}.  ACM
SIGCOMM, August 2013.
\end{sphinxadmonition}


\section{4.4 Architect to Evolve}
\label{\detokenize{ran:architect-to-evolve}}
\sphinxAtStartPar
We conclude this description of RAN internals by re\sphinxhyphen{}visiting the
sequence of steps involved in disaggregation, which as the previous
three sections reveal, is being pursued in multiple tiers. In doing
so, we tie up several loose ends, including the new interfaces
disaggregation exposes. These interfaces define the pivot points
around which 5G RAN is architected to evolve.

\sphinxAtStartPar
In the first tier of disaggregation, 3GPP standards provide multiple
options of how horizontal RAN splits can take place. Horizontal
disaggregation basically splits the RAN pipeline shown in
\hyperref[\detokenize{ran:fig-pipeline}]{Figure \ref{\detokenize{ran:fig-pipeline}}} into independently operating
components.  \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (a)} illustrates
horizontal disaggregation of the RAN from a single base station into
three distinct components: CU, DU and RU. The O\sphinxhyphen{}RAN Alliance has
selected specific disaggregation options from 3GPP and is developing
open interfaces between these components.  3GPP defines the \sphinxstylestrong{N2} and
\sphinxstylestrong{N3} interfaces between the RAN and the Mobile Core.

\sphinxAtStartPar
The second tier of disaggregation is vertical, focusing on
control/user plane separation (CUPS) of the CU, and resulting in CU\sphinxhyphen{}U
and CU\sphinxhyphen{}C shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (b)}. The control
plane in question is the 3GPP control plane, where the CU\sphinxhyphen{}U realizes a
pipeline for user traffic and the CU\sphinxhyphen{}C focuses on control message
signaling between Mobile Core and the disaggregated RAN components (as
well as to the UE). The O\sphinxhyphen{}RAN specified interfaces between these
disaggregated components are also shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (b)}.

\sphinxAtStartPar
The third tier follows the SDN paradigm by carrying vertical
disaggregation one step further. It does this by separating most of
RAN control (RRM functions) from the disaggregated RAN components, and
logically centralizing them as applications running on an SDN
Controller, which corresponds to the Near\sphinxhyphen{}RT RIC shown previously in
\hyperref[\detokenize{ran:fig-rrc-split}]{Figures \ref{\detokenize{ran:fig-rrc-split}}} and \hyperref[\detokenize{ran:fig-ran-controller}]{\ref{\detokenize{ran:fig-ran-controller}}}. This SDN\sphinxhyphen{}based vertical disaggregation is
repeated here in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (c)}.  The figure
also shows the additional O\sphinxhyphen{}RAN prescribed interfaces.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide39}.png}
\caption{Three tiers of RAN disaggregation: (a) horizontal, (b) vertical
CUPS, and (c) vertical SDN.}\label{\detokenize{ran:id6}}\label{\detokenize{ran:fig-disagg}}\end{figure}

\sphinxAtStartPar
The interface names are cryptic, and knowing their details adds little
to our conceptual understanding of the RAN, except perhaps to
re\sphinxhyphen{}enforce how challenging it is to introduce a transformative
technology like Software\sphinxhyphen{}Defined Networking into an operational
environment that is striving to achieve full backward compatibility
and universal interoperability. That said, we call out two notable
examples.

\sphinxAtStartPar
The first is the \sphinxstylestrong{A1} interface that the mobile operator’s
management plane—typically called the \sphinxstyleemphasis{OSS/BSS (Operations Support
System / Business Support System)} in the Telco world—uses to
configure the RAN.  We have not discussed the Telco OSS/BSS up to this
point, but it safe to assume such a component sits at the top of any
Telco software stack. It is the source of all configuration settings
and business logic needed to operate a network. Notice that the
Management Plane shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}} (c)}
includes a \sphinxstyleemphasis{Non\sphinxhyphen{}Real\sphinxhyphen{}Time RIC} functional block, complementing the
Near\sphinxhyphen{}RT RIC that sits below the A1 interface. We return to the
relevance of these two RICs in a moment.

\sphinxAtStartPar
The second is the \sphinxstylestrong{E2} interface that the Near\sphinxhyphen{}RT RIC uses to
control the underlying RAN elements. A requirement of the E2 interface
is that it be able to connect the Near\sphinxhyphen{}RT RIC to different types of
RAN elements. This range is reflected in the API, which revolves
around a \sphinxstyleemphasis{Service Model} abstraction. The idea is that each RAN
element advertises a Service Model, which effectively defines the set
of RAN Functions the element is able to support. The RIC then issues a
combination of the following four operations against this Service
Model.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Report:} RIC asks the element to report a function\sphinxhyphen{}specific value setting.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Insert:} RIC instructs the element to activate a user plane function.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Control:} RIC instructs the element to activate a control plane function.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Policy:} RIC sets a policy parameter on one of the activated functions.

\end{itemize}

\sphinxAtStartPar
Of course, it is the RAN element, through its published Service Model,
that defines the relevant set of functions that can be activated, the
variables that can be reported, and policies that can be set.

\sphinxAtStartPar
Taken together, the A1 and E2 interfaces complete two of the three
major control loops of the RAN: the outer (non\sphinxhyphen{}real\sphinxhyphen{}time) loop has the
Non\sphinxhyphen{}RT RIC as its control point and the middle (near\sphinxhyphen{}real\sphinxhyphen{}time) loop has
the Near\sphinxhyphen{}RT RIC as its control point. The third (inner) control loop,
which is not shown in \hyperref[\detokenize{ran:fig-disagg}]{Figure \ref{\detokenize{ran:fig-disagg}}}, runs inside
the DU: It includes the real\sphinxhyphen{}time Scheduler embedded in the MAC stage
of the RAN pipeline. The two outer control loops have rough time
bounds of \textgreater{}\textgreater{}1sec and \textgreater{}10ms, respectively, and as we saw in Chapter 2,
the real\sphinxhyphen{}time control loop is assumed to be \textless{}1ms.

\sphinxAtStartPar
This raises the question of how specific functionality is distributed
between the Non\sphinxhyphen{}RT RIC, Near\sphinxhyphen{}RT RIC, and DU. Starting with the second
pair (i.e., the two inner loops), it is important to recognize that
not all RRM functions can be centralized. After horizontal and
vertical CUPS disaggregation, the RRM functions are split between CU\sphinxhyphen{}C
and DU. For this reason, the SDN\sphinxhyphen{}based vertical disaggregation focuses
on centralizing CU\sphinxhyphen{}C\sphinxhyphen{}side RRM functions in the Near\sphinxhyphen{}RT RIC. In
addition to RRM control, this includes all the SON applications.

\sphinxAtStartPar
Turning to the outer two control loops, the Near RT\sphinxhyphen{}RIC opens the
possibility of introducing policy\sphinxhyphen{}based RAN control, whereby
interrupts (exceptions) to operator\sphinxhyphen{}defined policies would signal the
need for the outer loop to become involved. For example, one can
imagine developing learning\sphinxhyphen{}based controls, where the inference
engines for these controls would run as applications on the Near
RT\sphinxhyphen{}RIC, and their non\sphinxhyphen{}real\sphinxhyphen{}time learning counterparts would run
elsewhere. The Non\sphinxhyphen{}RT RIC would then interact with the Near\sphinxhyphen{}RT RIC to
deliver relevant operator policies from the Management Plane to the
Near RT\sphinxhyphen{}RIC over the A1 interface.

\sphinxAtStartPar
Finally, you may be wondering why there is an O\sphinxhyphen{}RAN Alliance in the
first place, given that 3GPP is already the standardization body
responsible for interoperability across the global cellular network.
The answer is that over time 3GPP has become a vendor\sphinxhyphen{}dominated
organization, whereas O\sphinxhyphen{}RAN was created more recently by network
operators. (AT\&T and China Mobile were the founding members.) O\sphinxhyphen{}RAN’s
goal is to catalyze a software\sphinxhyphen{}based implementation that breaks the
vendor lock\sphinxhyphen{}in that dominates today’s marketplace. The E2 interface
in particular, which is architected around the idea of supporting
different Service Models, is central to this strategy. Whether the
operators will be successful in their ultimate goal is yet to be seen.


\chapter{Chapter 5:  Advanced Capabilities}
\label{\detokenize{disaggregate:chapter-5-advanced-capabilities}}\label{\detokenize{disaggregate::doc}}
\sphinxAtStartPar
Disaggregating the cellular network pays dividends. This chapter
explores three examples. Stepping back to look at the big picture,
Chapter 3 (Architecture) described “what is” (the basics of 3GPP) and
Chapter 4 (RAN Internals) described “what will be” (where the industry
is clearly headed), whereas this chapter describes “what might be”
(our best judgement on cutting\sphinxhyphen{}edge capabilities that will eventually
be realized).


\section{5.1 Optimized Data Plane}
\label{\detokenize{disaggregate:optimized-data-plane}}
\sphinxAtStartPar
There are many reasons to disaggregate functionality, but one of the
most compelling is that decoupling control and data code paths allows
them to be optimized independently. The data path, for example, can be
optimized by
programming it into specialized hardware. Modern white\sphinxhyphen{}box switches with
programmable packet forwarding pipelines are a perfect example of
specialized hardware we can exploit in the cellular network.
\hyperref[\detokenize{disaggregate:fig-e2e}]{Figure \ref{\detokenize{disaggregate:fig-e2e}}} shows the first step in the process
of doing this. The figure also pulls together all the elements we’ve
been describing up to this point.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide21}.png}
\caption{End\sphinxhyphen{}to\sphinxhyphen{}end disaggregated system, including Mobile Core
and Split\sphinxhyphen{}RAN.}\label{\detokenize{disaggregate:id1}}\label{\detokenize{disaggregate:fig-e2e}}\end{figure}

\sphinxAtStartPar
There are several things to note
about this diagram. First, the figure combines both the Mobile Core and RAN elements,
organized according to the major subsystems: Mobile Core, Central Unit
(CU), Distributed Unit (DU), and Radio Unit (RU). The figure also shows
one possible mapping of these subsystems onto physical locations, with
the first two co\sphinxhyphen{}located in a Central Office and the latter two
co\sphinxhyphen{}located in a cell tower. As discussed earlier, other configurations
are also possible.

\sphinxAtStartPar
Second, the figure shows the Mobile Core’s two user plane elements
(PGW, SGW) and the Central Unit’s single user plane element (PDCP)
further disaggregated into control/user plane pairs, denoted PGW\sphinxhyphen{}C /
PGW\sphinxhyphen{}U, SGW\sphinxhyphen{}C / SGW\sphinxhyphen{}U, and PDCP\sphinxhyphen{}C / PDCP\sphinxhyphen{}U, respectively. Exactly how
this decoupling is realized is an implementation choice (i.e., not
specified by 3GPP), but the idea is to reduce the User Plane component
to the minimal Receive\sphinxhyphen{}Packet / Process\sphinxhyphen{}Packet / Send\sphinxhyphen{}Packet
processing core, and elevate all control\sphinxhyphen{}related aspects into the
Control Plane component.

\sphinxAtStartPar
Third, the PHY (Physical) element of the RAN pipeline is split between
the DU and RU partition. Although beyond the scope of this book, the
3GPP spec specifies the PHY element as a collection of functional
blocks, some of which can be effectively implemented by software running
on a general\sphinxhyphen{}purpose processor and some of which are best implemented in
specialized hardware (e.g., a Digital Signal Processor). These two
subsets of functional blocks map to the PHY Upper (part of the DU) and
the PHY Lower (part of the RU), respectively.

\sphinxAtStartPar
Fourth, and somewhat confusingly, \hyperref[\detokenize{disaggregate:fig-e2e}]{Figure \ref{\detokenize{disaggregate:fig-e2e}}}
shows the PDCP\sphinxhyphen{}C
element and the Control Plane (Forwarding) element combined into a
single functional block, with a data path (blue line) connecting that
block to both the RLC and the MME. Exactly how this pair is realized is
an implementation choice (e.g., they could map onto two or more
microservices), but the end result is that they are part of an
end\sphinxhyphen{}to\sphinxhyphen{}end path over which the MME can send control packets to the UE.
Note that this means responsibility for demultiplexing incoming packets
between the control plane and user plane falls to the RLC.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide22}.png}
\caption{Implementing data plane elements of the User Plane in
programmable switches.}\label{\detokenize{disaggregate:id2}}\label{\detokenize{disaggregate:fig-e2e-p4}}\end{figure}

\sphinxAtStartPar
\hyperref[\detokenize{disaggregate:fig-e2e-p4}]{Figure \ref{\detokenize{disaggregate:fig-e2e-p4}}} shows why we disaggregated these
components: it allows us to realize the three user plane elements
(PGW\sphinxhyphen{}U, SGW\sphinxhyphen{}U, PDCP\sphinxhyphen{}U) in switching hardware. This can be done using a
combination of a language that is tailored for programming forwarding
pipelines (e.g., P4), and a protocol\sphinxhyphen{}independent switching
architecture (e.g., Intel’s Tofino). For now, the important takeaway is that
the RAN and Mobile Core user plane can be mapped directly onto an
SDN\sphinxhyphen{}enabled data plane.

\phantomsection\label{\detokenize{disaggregate:reading-p4}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For more information about P4 and programmable switching chips, we
recommend \sphinxhref{https://sdn.systemsapproach.org/switch.html}{White\sphinxhyphen{}Box Switches}, a chapter in
\sphinxstyleemphasis{Software\sphinxhyphen{}Defined Networking: A Systems Approach}, March 2020.
\end{sphinxadmonition}

\sphinxAtStartPar
Pushing RAN and Mobile Core forwarding functionality into the switching
hardware results in overlapping terminology that can be confusing.
5G separates the functional blocks into control and user planes,
while SDN disaggregates a given functional block into control and data
plane halves. The overlap comes from our choosing to implement
the 5G user plane by splitting it into its SDN\sphinxhyphen{}based control and data
plane parts. As one simplification, we refer to the Control Plane
(Forwarding) and PDCP\sphinxhyphen{}C combination as the CU\sphinxhyphen{}C (Central Unit \sphinxhyphen{} Control)
going forward.

\sphinxAtStartPar
Finally, the SDN\sphinxhyphen{}prescribed control/data plane disaggregation comes with
an implied implementation strategy, namely, the use of a scalable and
highly available \sphinxstyleemphasis{Network Operating System (NOS)}. Like a traditional
OS, a NOS sits between application programs (control apps) and the
underlying hardware devices (whitebox switches), providing higher levels
abstractions (e.g., network graph) to those applications, while hiding
the low\sphinxhyphen{}level details of the underlying hardware. To make the discussion
more concrete, we use ONOS (Open Network Operating System) as an example
NOS, where PGW\sphinxhyphen{}C, SGW\sphinxhyphen{}C, and PDCP\sphinxhyphen{}C are all realized as control
applications running on top of ONOS.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=400\sphinxpxdimen]{{Slide23}.png}
\caption{Control Plane elements of the User Plane implemented
as Control Applications running on an SDN Controller (e.g., ONOS).}\label{\detokenize{disaggregate:id3}}\label{\detokenize{disaggregate:fig-onos}}\end{figure}

\sphinxAtStartPar
\hyperref[\detokenize{disaggregate:fig-onos}]{Figure \ref{\detokenize{disaggregate:fig-onos}}} shows one possible configuration in
which the underlying switches are interconnected to form a leaf\sphinxhyphen{}spine
fabric. Keep in mind that the linear sequence of switches implied by
\hyperref[\detokenize{disaggregate:fig-e2e-p4}]{Figure \ref{\detokenize{disaggregate:fig-e2e-p4}}} is logical, but that in no way
restricts the actual hardware to the same topology. The reason we use
a leaf\sphinxhyphen{}spine topology is related to our ultimate goal of building an
edge cloud, and leaf\sphinxhyphen{}spine is the prototypical structure for such
cloud\sphinxhyphen{}based clusters.  This means the three control applications must
work in concert to implement an end\sphinxhyphen{}to\sphinxhyphen{}end path through the fabric,
which in practice happens with the aid of other, fabric\sphinxhyphen{}aware, control
applications (as implied by the “…” in the Figure). We describe the
complete picture in more detail in the next chapter, but for now, the
big takeaway is that the control plane components of the 5G overlay
can be realized as control applications for an SDN\sphinxhyphen{}based underlay.


\section{5.2 Multi\sphinxhyphen{}Cloud}
\label{\detokenize{disaggregate:multi-cloud}}
\sphinxAtStartPar
Another consequence of disaggregating functionality is that once
decoupled, different functions can be placed in different physical
locations. We have already seen this when we split the RAN, placing some
functions (e.g., the PDCP and RRC) in the Central Unit and others (e.g.,
RLC and MAC) in Distributed Units. This allows for simpler (less
expensive) hardware in remote locations, where there are often space,
power, and cooling constraints.

\sphinxAtStartPar
This process can be repeated by distributing the more centralized
elements across multiple clouds, including large datacenters that
already benefit from elasticity and economies of
scale. \hyperref[\detokenize{disaggregate:fig-multicloud}]{Figure \ref{\detokenize{disaggregate:fig-multicloud}}} shows the resulting
multi\sphinxhyphen{}cloud realization of the Mobile Core. We leave the user plane at
the edge of the network (e.g., in the Central Office) and move control
plane to a centralized cloud. It could even be a public cloud like
Google or Amazon. This includes not only the MME, PCRF, and HSS, but
also the PGW\sphinxhyphen{}C and SGW\sphinxhyphen{}C we decoupled in the previous section. (Note
that \hyperref[\detokenize{disaggregate:fig-multicloud}]{Figure \ref{\detokenize{disaggregate:fig-multicloud}}} renames the PDCP\sphinxhyphen{}U from
earlier diagrams as the CU\sphinxhyphen{}U; either label is valid.)

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide24}.png}
\caption{Multi\sphinxhyphen{}Cloud implementation, with MME, HSS, PCRF and
Control Plane elements of the PGW and SGW running in a centralized
cloud.}\label{\detokenize{disaggregate:id4}}\label{\detokenize{disaggregate:fig-multicloud}}\end{figure}

\sphinxAtStartPar
What is the value in doing this? Just like the DU and RU, the Edge Cloud
likely has limited resources. If we want room to run new edge services
there, it helps to move any components that need not be local to a
larger facility with more abundant resources. Centralization also
facilitates collecting and analyzing data across multiple edge
locations, which is harder to do if that information is distributed over
multiple sites. (Analytics performed on this data also benefits from
having abundant compute resources available.)

\sphinxAtStartPar
Another reason worth calling out is that it lowers the barrier for
anyone (not just the companies that own and operate the RAN
infrastructure) to offer mobile services to customers. These entities
are called \sphinxstyleemphasis{MVNOs (Mobile Virtual Network Operators)} and one clean
way to engineer an MVNO is to run your own Mobile Core in a cloud of
your choosing.

\sphinxAtStartPar
But the biggest motivation for the configuration shown in
\hyperref[\detokenize{disaggregate:fig-multicloud}]{Figure \ref{\detokenize{disaggregate:fig-multicloud}}} is that keeping the user plane
elements of the mobile core at the edge makes it possible to “break
out” local traffic without having to traverse a “hairpin” route that
goes through a central site. This has the potential to dramatically
reduce latency for any edge\sphinxhyphen{}hosted services. We return to this topic
in Chapter 7.


\section{5.3 Network Slicing}
\label{\detokenize{disaggregate:network-slicing}}
\sphinxAtStartPar
One of the most compelling value propositions of 5G is the ability to
differentiate the level of service offered to different applications
and customers. Differentiation, of course, is key to being able to
charge some customers more than others, but the monetization case
is only part of the story. It is also necessary to support
applications with widely varying requirements. For example, streaming video requires high bandwidth
but can tolerate larger latencies, while IoT has
minimal bandwidth needs but sometimes requires extremely low and predictable
latencies, and entails connecting a \sphinxstyleemphasis{massively scalable} number of
devices.

\sphinxAtStartPar
The mechanism that supports this sort of differentiation is called
network slicing, and it fundamentally comes down to scheduling, both in
the RAN (deciding which segments to transmit) and in the Mobile Core
(scaling microservice instances and placing those instances on the
available servers). The following introduces the basic idea, starting
with the RAN.

\sphinxAtStartPar
But before getting into the details, we note that a network slice is a
realization of the QoS Class Identifier (QCI) discussed earlier. 3GPP
specifies a standard set of network slices, called \sphinxstyleemphasis{Standardized Slice
Type (SST)} values. For example, SST 1 corresponds to mobile broadband,
SST 2 corresponds to Ultra\sphinxhyphen{}Reliable Low Latency Communications, SST 3
corresponds to Massive IoT, and so on. It is also possible to extend
this standard set with additional slice behaviors, as well as define
multiple slices for each SST (e.g., to further differentiate subscribers
based on priority).

\sphinxAtStartPar
Of course, defining a desired set of slices and implementing a slicing
mechanism are two entirely different things. The following sketches
how slices can be implemented.


\subsection{RAN Slicing}
\label{\detokenize{disaggregate:ran-slicing}}
\sphinxAtStartPar
We start by reviewing the basic scheduling challenge previewed in
Chapter 2. As depicted in \hyperref[\detokenize{disaggregate:fig-slice-sched}]{Figure \ref{\detokenize{disaggregate:fig-slice-sched}}},
the radio spectrum can be conceptualized as a 2\sphinxhyphen{}D grid of
\sphinxstyleemphasis{Resource Blocks (RB)}, where the scheduler’s job is to decide how to fill the
grid with the available segments from each user’s transmission queue
based on CQI feedback from the UEs. To restate, the power of OFDMA is
the flexibility it provides in how this mapping is performed.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=450\sphinxpxdimen]{{Slide27}.png}
\caption{Scheduler allocating resource blocks to UEs.}\label{\detokenize{disaggregate:id5}}\label{\detokenize{disaggregate:fig-slice-sched}}\end{figure}

\sphinxAtStartPar
While in principle one could define an uber scheduler that takes dozens
of different factors into account, the key to network slicing is to add
a layer of indirection. As shown in \hyperref[\detokenize{disaggregate:fig-hypervisor}]{Figure \ref{\detokenize{disaggregate:fig-hypervisor}}},
the idea is to perform a second mapping of Virtual RBs to
Physical RBs. This sort of virtualization is common in resource
allocators throughout computing systems because we want to separate how
many resources are allocated to each user from the decision as to which
physical resources are actually assigned. This virtual\sphinxhyphen{}to\sphinxhyphen{}physical
mapping is performed by a layer typically known as a \sphinxstyleemphasis{Hypervisor}, and
the important thing to keep in mind is that it is totally agnostic about
which user’s segment is affected by each translation.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide28}.png}
\caption{Wireless Hypervisor mapping virtual resource blocks to
physical resource blocks.}\label{\detokenize{disaggregate:id6}}\label{\detokenize{disaggregate:fig-hypervisor}}\end{figure}

\sphinxAtStartPar
Having decoupled the Virtual RBs from Physical RBs, it is now possible
to define multiple Virtual RB sets (of varying sizes), each with its own
scheduler. \hyperref[\detokenize{disaggregate:fig-multi-sched}]{Figure \ref{\detokenize{disaggregate:fig-multi-sched}}} gives an example with two
equal\sphinxhyphen{}sized RB sets, where the important consequence is that having made
the macro\sphinxhyphen{}decision that the Physical RBs are divided into two equal
partitions, the scheduler associated with each partition is free to
allocate Virtual RBs completely independent from each other. For
example, one scheduler might be designed to deal with high\sphinxhyphen{}bandwidth
video traffic and another scheduler might be optimized for low\sphinxhyphen{}latency
IoT traffic. Alternatively, a certain fraction of the available capacity
could be reserved for premium customers or other high\sphinxhyphen{}priority traffic
(e.g., public safety), with the rest shared among everyone else.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide29}.png}
\caption{Multiple schedulers running on top of wireless
hypervisor.}\label{\detokenize{disaggregate:id7}}\label{\detokenize{disaggregate:fig-multi-sched}}\end{figure}

\sphinxAtStartPar
Going one level deeper in the implementation details, the real\sphinxhyphen{}time
scheduler running in each DU receives high\sphinxhyphen{}level directives from the
near\sphinxhyphen{}real\sphinxhyphen{}time scheduler running in the CU, and as depicted in
\hyperref[\detokenize{disaggregate:fig-slicing-control}]{Figure \ref{\detokenize{disaggregate:fig-slicing-control}}}, the DUs follow these
directives in making their scheduling decisions \sphinxstyleemphasis{on a
per\sphinxhyphen{}slice basis}. A single RAN Slicing control application is responsible for the
macro\sphinxhyphen{}scheduling decision by allocating resources among a set of
slices. Understanding this implementation detail is important because
all of these control decisions are implemented by software modules,
and hence, easily changed or customized. They are not “locked” into
the underlying system, as they have historically been in 4G’s eNodeBs.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=350\sphinxpxdimen]{{Slide30}.png}
\caption{Centralized near\sphinxhyphen{}real\sphinxhyphen{}time control applications
cooperating with distributed real\sphinxhyphen{}time RAN schedulers.}\label{\detokenize{disaggregate:id8}}\label{\detokenize{disaggregate:fig-slicing-control}}\end{figure}

\sphinxAtStartPar
In summary, the goal of RAN slicing is to programmatically create
virtual RAN nodes (base stations) that operate on the same hardware
and share the spectrum resources according to a given policy for
different applications, services, users, and so on. Tying RAN slicing
back to RAN disaggregation, one can imagine several possible
configurations, depending on the desired level of isolation between
the slices. \hyperref[\detokenize{disaggregate:fig-ran-slicing}]{Figure \ref{\detokenize{disaggregate:fig-ran-slicing}}} shows four examples,
all of which assume slices share the antennas and RF components, which
is effectively the RU: (a) RAN slices share RU, DU, CU\sphinxhyphen{}U and CU\sphinxhyphen{}C; (b)
RAN slices share RU and DU, but have their own CU\sphinxhyphen{}U and CU\sphinxhyphen{}C; (c) RAN
slices share RU, CU\sphinxhyphen{}U and CU\sphinxhyphen{}C, but have their own DU; and (d) RAN
slices share RU, but have their own DU, CU\sphinxhyphen{}U and CU\sphinxhyphen{}C.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide40}.png}
\caption{Four possible configurations of a disaggregated RAN in support of
RAN slicing.}\label{\detokenize{disaggregate:id9}}\label{\detokenize{disaggregate:fig-ran-slicing}}\end{figure}


\subsection{Core Slicing}
\label{\detokenize{disaggregate:core-slicing}}
\sphinxAtStartPar
In addition to slicing the RAN, we also need to slice the Mobile Core.
In many ways, this is a well\sphinxhyphen{}understood problem, involving QoS
mechanisms in the network switches (i.e., making sure packets flow
through the switching fabric according to the bandwidth allocated to
each slice) and the cluster processors (i.e., making sure the containers
that implement each microservice are allocated sufficient CPU cores to
sustain the packet forwarding rate of the corresponding slice).

\sphinxAtStartPar
But packet scheduling and CPU scheduling are low\sphinxhyphen{}level mechanisms. What
makes slicing work is to also virtualize and replicate the entire
service graph that implements the Mobile Core. If you think of a slice as
a system abstraction, then that abstraction needs to keep track of the
set of interconnected microservices that implement each slice,
and then instruct the underlying packet schedulers to allocate
sufficient network bandwidth to the slice’s flows and the underlying CPU
schedulers to allocate sufficient compute cycles to the slice’s
containers.

\sphinxAtStartPar
For example, if there are two network slices (analogous to the two RAN
schedulers shown in \hyperref[\detokenize{disaggregate:fig-multi-sched}]{Figures \ref{\detokenize{disaggregate:fig-multi-sched}}} and
\hyperref[\detokenize{disaggregate:fig-slicing-control}]{\ref{\detokenize{disaggregate:fig-slicing-control}}}), then there would also need
to be two Mobile Core service graphs: One set of AMF, SMF, UPF,…
microservices running on behalf of the first slice and a second set of
AMF, SMF, UPF,… microservices running on behalf of the second
slice. These two graphs would scale independently (i.e., include a
different number of container instances), depending on their
respective workloads and QoS guarantees.  The two slices would also be
free to make different implementation choices, for example, with one
optimized for massive IoT applications and the other optimized for
high\sphinxhyphen{}bandwidth AR/VR applications.

\sphinxAtStartPar
The one remaining mechanism we need is a demultiplexing function that
maps a given packet flow (e.g., between UE and some Internet
application) onto the appropriate instance of the service graph. This is
the job of the NSSF described in an Chapter 3: it is responsible
for selecting the instance a given slice’s traffic is to traverse.

\sphinxAtStartPar
The similarity between slicing and the much\sphinxhyphen{}debated topic of network
QoS might lead one to conclude that slicing won’t take off, as QoS
never seemed to provide quite enough benefit in large networks to
justify its complexity. But the one place where QoS \sphinxstyleemphasis{is} effective is
when bandwidth is scarce, e.g., in last\sphinxhyphen{}mile links. For this reason,
slicing is likely to be valuable precisely because it allows efficient
partitioning of the relatively scarce resource that is cellular
spectrum.

\sphinxAtStartPar
Finally, note that slicing can be viewed as a way to enable overlays on
the cellular network. The Internet has a long history of supporting
innovation through the use of overlays. Many innovations such as
multicast and content delivery networks were developed in this way. So
while we might not know exactly how slicing will be used, it is an
important ingredient in fostering innovation.


\chapter{Chapter 6:  Exemplar Implementation}
\label{\detokenize{impl:chapter-6-exemplar-implementation}}\label{\detokenize{impl::doc}}
\sphinxAtStartPar
The steps we’ve taken in the previous chapters to virtualize,
disaggregate, optimize, distribute, and slice the cellular network not
only help us understand the inner\sphinxhyphen{}workings of 5G, but they are also
necessary to reduce the entirety of the 5G cellular network to practice.
The goal is an implementation, which by definition, forces us to make
specific engineering choices. This chapter describes one set of
engineering choices that results in a running system. It should be
interpreted as an exemplar, for the sake of completeness, but not the
only possibility.

\sphinxAtStartPar
The system we describe is called CORD, which you will recall from the
Introduction is an acronym (\sphinxstylestrong{C}entral \sphinxstylestrong{O}ffice
\sphinxstylestrong{R}e\sphinxhyphen{}architected as a \sphinxstylestrong{D}atacenter). More concretely, CORD is a
blueprint for building a 5G deployment from commodity hardware and a
collection of open source software components. We call this
hardware/software combination a CORD POD, where the idea is to deploy a
POD at each edge site that is part of a cellular network. The following
describes CORD in terms of a set of engineering decisions. It is not a
substitute for detailed documentation for installing, developing, and
operating CORD. Also keep in mind that even though CORD includes
“Central Office” in its name, a CORD POD is a general design, and not
strictly limited to being deployed in a conventional Central Office.

\phantomsection\label{\detokenize{impl:reading-guide}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
To learn how to install, operate, and contribute to the CORD open
source software platform, see the \sphinxhref{https://guide.opencord.org}{CORD Guide}. ONF, March 2020.
\end{sphinxadmonition}

\sphinxAtStartPar
Before getting into the specifics, it is important to understand that
CORD is a work\sphinxhyphen{}in\sphinxhyphen{}progress, with a sizable open source community
contributing to its code base. Many of the components are quite mature,
and currently running in operator trials and production networks. Others
(largely corresponding to the advanced capabilities described in the
previous chapter) are prototypes that run in “demonstration mode,” but
are not yet complete enough to be included in official releases. Also,
as outlined in the earlier discussion on deployment options, CORD starts
with a production\sphinxhyphen{}quality EPC that is being incrementally evolved into
its 5G counterpart. (This chapter uses the EPC\sphinxhyphen{}specific components for
illustrative purposes.)


\section{6.1 Framework}
\label{\detokenize{impl:framework}}
\sphinxAtStartPar
\hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} gives a schematic overview of a CORD POD. It
connects downstream to a set of DUs (and associated RUs), and upstream
to the rest of the Internet. Internally, it includes a set of commodity
servers (the figure shows four racks of three servers each, but the
design accommodates anywhere from a partial rack to 16 racks) and a set
of white\sphinxhyphen{}box switches arranged in a leaf\sphinxhyphen{}spine topology (the figure
shows two leaves and two spine switches, but the design allows anywhere
from a single switch to two leaf switches per rack and as many spine
switches as necessary to provide sufficient east\sphinxhyphen{}to\sphinxhyphen{}west bandwidth).

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=650\sphinxpxdimen]{{Slide25}.png}
\caption{CORD implementation of RAN and Mobile Core.}\label{\detokenize{impl:id1}}\label{\detokenize{impl:fig-cord}}\end{figure}

\sphinxAtStartPar
With respect to software, \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} shows a
combination of RAN (red) and Mobile Core (blue) components, plus the
modules that define the CORD platform (orange). We describe the
platform components later in this chapter, but you can think of them
as collectively implementing a multi\sphinxhyphen{}tenant cloud on top of which many
different scalable services can run. The RAN and Mobile Core are two
such tenants.  The CORD platform can also host other edge services
(which is one reason CORD is built using cloud technology in the first
place), but exactly what other edge services might run on a given CORD
POD is a question we do not try to answer in this book.

\sphinxAtStartPar
The RAN and Core related components are ones we’ve described in
earlier chapters. They include the Control and User planes of the CU
and Mobile Core, respectively. To simplify the diagram, we show
the SGW and PGW merged into a single S/PGW. One other detail that
deserves closer attention is the RAN Control component included in the
CU Control Plane.  This is the Near\sphinxhyphen{}RT RIC introduced in Section 4.3,
which means a CORD POD includes two SDN Controllers: the RIC controls
the RAN and ONOS (shown in \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}}) controls the
fabric.  (The RIC running in CORD actually happens to be a second,
customized version of ONOS, but that’s an implementation detail.)

\sphinxAtStartPar
One aspect of \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} that requires further
elaboration is how each of the RAN and Mobile Core components are
actually realized. Specifically, there are three different
manifestations of the functional components implied by the figure: (1)
the data plane layer of the CU\sphinxhyphen{}U and S/PGW\sphinxhyphen{}U are realized as P4
programs loaded into the programmable switches; (2) the control plane
layer of the CU\sphinxhyphen{}U and S/PGW\sphinxhyphen{}U (as well as the Trellis platform module)
are realized as control applications loaded onto ONOS;
and (3) the remaining components are realized as Kubernetes\sphinxhyphen{}managed
microservices. (Kubernetes is implied, but not shown in the figure.)

\sphinxAtStartPar
To expand on this idea, \hyperref[\detokenize{impl:fig-ci-cd}]{Figure \ref{\detokenize{impl:fig-ci-cd}}} gives an
alternative view of a CORD POD, abstracting away the details of \sphinxstyleemphasis{what}
services it hosts, and focusing instead on \sphinxstyleemphasis{how} those services are
instantiated on the POD. In this figure, all the functionality
instantiated onto the POD runs as a combination of Kubernetes\sphinxhyphen{}based
microservices and ONOS\sphinxhyphen{}based control applications.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=350\sphinxpxdimen]{{Slide26}.png}
\caption{Alternative view of CORD, with a CI/CD toolchain
managing the platform and set of services implemented by a
combination of ONOS\sphinxhyphen{}based control apps and Kubernetes\sphinxhyphen{}based
microservices.}\label{\detokenize{impl:id2}}\label{\detokenize{impl:fig-ci-cd}}\end{figure}

\sphinxAtStartPar
When abstracted in this way, we can view a POD as including three major
subsystems.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Platform:} The base layer common to all configurations includes
Kubernetes as the container management system and ONOS as the SDN
controller, with Stratum (an open source switch OS) loaded on to each switch. Both ONOS and the
control applications it hosts run as Kubernetes\sphinxhyphen{}managed
microservices.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Profile:} The deployment\sphinxhyphen{}specific collection of microservices and
SDN control apps that have been selected to run on a particular POD.
This is a variable and evolvable set, and it includes the control
plane and edge services described elsewhere.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{CI/CD Toolchain:} Used to assemble, deploy, operate, and upgrade a
particular Platform/Profile combination. It implements a set of
processes that transforms a collection of disaggregated and
virtualized components into an operational system capable of
responding to operator directives and carrying live traffic.

\end{itemize}

\sphinxAtStartPar
Although beyond the scope of this book, the CI/CD toolchain uses
standard DevOps tools to bootstrap software onto the cluster of servers
and switches, and rollout/rollback individual microservices and control
applications. It also auto\sphinxhyphen{}generates the Northbound Interface (NBI) that
operators use to manage the POD, based on a declarative specification of
the Profile the POD is configured to support. This NBI is sufficiently
complete to operate a CORD POD in a production environment.


\section{6.2 Platform Components}
\label{\detokenize{impl:platform-components}}
\sphinxAtStartPar
We now return to the three platform\sphinxhyphen{}related components shown in
\hyperref[\detokenize{impl:fig-cord}]{Figures \ref{\detokenize{impl:fig-cord}}} and \hyperref[\detokenize{impl:fig-ci-cd}]{\ref{\detokenize{impl:fig-ci-cd}}}. Each is a substantial open source component in its own
right, but for our purposes, it is enough to understand the role they
play in supporting a 5G\sphinxhyphen{}based profile of CORD.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Stratum:} A thin operating system that runs locally on each
white\sphinxhyphen{}box switch. Its purpose is to provide a hardware\sphinxhyphen{}independent
interface for managing and programming the switches in CORD. This
includes using \sphinxstyleemphasis{P4} to define the forwarding behavior of the switch’s
forwarding pipeline (think of this program as a contract between the
control and data planes), and \sphinxstyleemphasis{P4Runtime} to control that forwarding
contract at runtime.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{ONOS:} A Network Operating System used to configure and control a
network of programmable white\sphinxhyphen{}box switches. It runs off\sphinxhyphen{}switch as a
logically centralized SDN controller, and hosts a collection of SDN
control applications, each of which controls some aspect of the
underlying network. Because it is logically centralized, ONOS is
designed to be highly available and to have scalable performance.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Trellis:} An ONOS\sphinxhyphen{}hosted SDN control application that implements a
leaf\sphinxhyphen{}spine fabric on a network of white\sphinxhyphen{}box switches. It implements
the control plane for several features, including VLANs and L2
bridging, IPv4 and IPv6 unicast and multicast routing, DHCP L3 relay,
dual\sphinxhyphen{}homing of servers and upstream routers, QinQ
forwarding/termination, MPLS pseudowires, and so on. In addition,
Trellis can make the entire fabric appear as a single (virtual)
router to upstream routers, which communicate with Trellis using
standard BGP.

\end{itemize}

\sphinxAtStartPar
Stratum (running on each switch) and ONOS (running off\sphinxhyphen{}switch and
managing a network of switches) communicate using the following
interfaces.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{P4:} Defines the forwarding behavior for programmable switching
chips as well as modeling fixed\sphinxhyphen{}function ASIC pipelines. A P4 program
defines a contract that is implemented by the data plane and
programmable by the control plane.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{P4Runtime:} An SDN\sphinxhyphen{}ready interface for controlling forwarding
behavior at runtime. It is the key for populating forwarding tables
and manipulating forwarding state, and it does so in way that is
hardware\sphinxhyphen{}agnostic.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{OpenConfig Models:} Define a base for device configuration and
management. These models can be programmatically extended for
platform\sphinxhyphen{}specific functionality, but the goal is to minimize model
deviations so as to enable a vendor\sphinxhyphen{}agnostic management plane.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{gNMI} (gRPC Network Management Interface): Improves on the
existing configuration interfaces by using a binary representation on
the wire and enabling bi\sphinxhyphen{}directional streaming. Paired with the
OpenConfig models, gNMI is SDN\sphinxhyphen{}ready.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{gNOI} (gRPC Network Operations Interfaces): A collection of
microservices that enable switch specific operations, like
certificate management, device testing, software upgrade, and
networking troubleshooting. gNOI provides a semantically rich API
that replaces existing CLI\sphinxhyphen{}based approaches.

\end{itemize}

\sphinxAtStartPar
Trellis, as an SDN control application running on top of ONOS, controls
packet forwarding across the switching fabric internal to a CORD POD
(i.e., within a single site). But Trellis can also be extended across
multiple sites deeper into the network using multiple stages of spines,
as shown in \hyperref[\detokenize{impl:fig-trellis}]{Figure \ref{\detokenize{impl:fig-trellis}}}. This means Trellis has the
potential to play a role in implementing the backhaul and midhaul
network for the RAN, or alternatively, extending the RAN into customer
premises (denoted “On Site” in the figure).

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=600\sphinxpxdimen]{{Slide31}.png}
\caption{Trellis control application managing a (possibly
distributed) leaf\sphinxhyphen{}spine fabric.}\label{\detokenize{impl:id3}}\label{\detokenize{impl:fig-trellis}}\end{figure}

\sphinxAtStartPar
The software stack we’ve just described is substantial, and has the
potential to disrupt the way cellular networks are built and operated.
Of particular note, the RAN Intelligent Controller
shown in \hyperref[\detokenize{impl:fig-cord}]{Figure \ref{\detokenize{impl:fig-cord}}} is implemented as a set of
extensions to ONOS.  This puts the ONOS\sphinxhyphen{}based RIC at the very center
of the design, where the SDN and 5G worlds intersect.

\sphinxAtStartPar
This discussion, while focused on just one option for implementing 5G
networks, illustrates one of the reasons 5G is viewed as such a
transformation of the Telco industry. The 5G architecture, far more
than any previous Telco network, capitalizes on several significant, broad
industry trends. These include the rise of SDN, the power of open source
software and its increasing use in networking, and of course the
adoption of cloud technologies as the basis for delivering innovative services.

\phantomsection\label{\detokenize{impl:reading-sdn}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For more information about the SDN software stack, we recommend a
companion book: \sphinxhref{https://sdn.systemsapproach.org/}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}. March 2020.
\end{sphinxadmonition}


\chapter{Chapter 7:  Cloudification of Access}
\label{\detokenize{cloud:chapter-7-cloudification-of-access}}\label{\detokenize{cloud::doc}}
\sphinxAtStartPar
The previous chapters went step\sphinxhyphen{}by\sphinxhyphen{}step, first breaking 5G down into its
elemental components and then showing how those components could be put
back together using best practices in cloud design to build a fully
functional, 3GPP\sphinxhyphen{}compliant 5G cellular network. In doing so, it is easy
to lose sight of the big picture, which is that the cellular network is
undergoing a dramatic transformation. That’s the whole point of 5G. We
conclude by making some observations about this big picture.


\section{7.1 Multi\sphinxhyphen{}Cloud}
\label{\detokenize{cloud:multi-cloud}}
\sphinxAtStartPar
To understand the impact of cloud technologies and practices being
applied to the access network, it is helpful to first understand what’s
important about the cloud. The cloud has fundamentally changed the way
we compute, and more importantly, the pace of innovation. It has done
this through a combination of the following.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Disaggregation:} Breaking vertically integrated systems into
independent components with open interfaces.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Virtualization:} Being able to run multiple independent copies of
those components on a common hardware platform.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Commoditization:} Being able to elastically scale those virtual
components across commodity hardware bricks as workload dictates.

\end{itemize}

\sphinxAtStartPar
There is an opportunity for the same to happen with the access network,
or from another perspective, for the cloud to essentially expand so far
as to subsume the access network.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide32}.png}
\caption{A collection of multi\sphinxhyphen{}tenant clouds—including virtualized RAN
resources alongside conventional compute, storage, and network
resources—hosting both Telco and Over\sphinxhyphen{}the\sphinxhyphen{}Top (OTT) services and
applications.}\label{\detokenize{cloud:id1}}\label{\detokenize{cloud:fig-cloud}}\end{figure}

\sphinxAtStartPar
\hyperref[\detokenize{cloud:fig-cloud}]{Figure \ref{\detokenize{cloud:fig-cloud}}} gives a high\sphinxhyphen{}level overview of how the
transformation might play out, with the global cloud spanning edge
clouds, private Telco clouds, and the familiar public clouds. We call
this collection of clouds “multi\sphinxhyphen{}cloud” (although note that there are
a number of other definitions for that term). Each
individual cloud site is potentially owned by a different organization
(this includes the cell towers, as well), and as a consequence, each
site will likely be multi\sphinxhyphen{}tenant in that it is able to host (and
isolate) applications on behalf of many other people and organizations.
Those applications, in turn, will include a combination of the RAN and
Core services (as described throughout this book), Over\sphinxhyphen{}the\sphinxhyphen{}Top (OTT)
applications commonly found today in public clouds (but now also
distributed across edge clouds), and new Telco\sphinxhyphen{}managed applications
(also distributed across centralized and edge locations).

\sphinxAtStartPar
Eventually, we can expect common APIs to emerge, lowering the barrier
for anyone (not just today’s network operators or cloud providers) to
deploy applications across multiple sites by acquiring the storage,
compute, networking, and connectivity resources they need.


\section{7.2 EdgeCloud\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service}
\label{\detokenize{cloud:edgecloud-as-a-service}}
\sphinxAtStartPar
Of all the potential outcomes discussed in the previous section, one
that is rapidly gaining traction is to run a 5G\sphinxhyphen{}enabled edge cloud as
a centrally managed service. As illustrated in \hyperref[\detokenize{cloud:fig-edgecloud}]{Figure \ref{\detokenize{cloud:fig-edgecloud}}}, the idea is to deploy an edge cloud in enterprises,
configured with the user plane components of the RAN and Mobile Core
(along with any edge services the enterprise wants to run locally),
and then manage that edge deployment from the central cloud. The
central cloud would run a management portal for the edge cloud, along
with the control plane of the Mobile Core. This is similar to the
multi\sphinxhyphen{}cloud configuration discussed in Section 5.2, except with the
added feature of being able to manage multiple edge deployments from
one central location.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide36}.png}
\caption{EdgeCloud\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service, a managed service, with RAN and Mobile
Core user plane components running in the enterprise, and the
control plane of the Mobile Core (along with a management portal)
running centrally in the public cloud.}\label{\detokenize{cloud:id2}}\label{\detokenize{cloud:fig-edgecloud}}\end{figure}

\sphinxAtStartPar
The value of such a deployment is to bring 5G wireless advantages into
the enterprise, including support for predictable, low\sphinxhyphen{}latency
communication required for real\sphinxhyphen{}time controlling of large numbers of
mobile devices. Factory automation is one compelling use case for such
an edge cloud, but interest in supporting IoT in general is giving
ECaaS significant momentum.

\sphinxAtStartPar
This momentum has, not surprisingly, led to recent commercial
activity. But there is also an open source variant, called Aether, now
available for early adopters to evaluate and experiment with. Aether
is an ONF\sphinxhyphen{}operated ECaaS with 4G/5G support, built from the open
source components described throughout this book. Aether works with
both licensed and unlicensed frequency bands (e.g., CBRS), but it is
the latter that makes it an easy system to opt into. \hyperref[\detokenize{cloud:fig-aether}]{Figure
\ref{\detokenize{cloud:fig-aether}}} depicts the early stages of Aether’s centrally
managed, multi\sphinxhyphen{}site deployment.

\begin{figure}[ht]
\centering
\capstart

\noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{Slide37}.png}
\caption{Aether is an ONF\sphinxhyphen{}operated EdgeCloud\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service built from the
SD\sphinxhyphen{}RAN and disaggregated Mobile Core components described
throughout this book. Aether includes a centralized operations
portal running in the Public Cloud.}\label{\detokenize{cloud:id3}}\label{\detokenize{cloud:fig-aether}}\end{figure}

\sphinxAtStartPar
Note that each edge site in \hyperref[\detokenize{cloud:fig-aether}]{Figure \ref{\detokenize{cloud:fig-aether}}}
corresponds to a CORD POD described in Chapter 6, re\sphinxhyphen{}configured to
offload the O\&M Interface and the Control elements of the Mobile Core
to the central cloud.

\phantomsection\label{\detokenize{cloud:reading-aether}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
For more information about Aether, visit the \sphinxhref{https://www.opennetworking.org/aether/}{Aether Web Site}. ONF, March 2020.
\end{sphinxadmonition}


\section{7.3 Research Opportunities}
\label{\detokenize{cloud:research-opportunities}}
\sphinxAtStartPar
In order for the scenarios described in this Chapter to become a
reality, a wealth of research problems need to be addressed, many of
which are a consequence of the blurring line between access networks
and the edge cloud. We refer to this as the \sphinxstyleemphasis{access\sphinxhyphen{}edge}, and we
conclude by identifying some example challenges/opportunities.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Multi\sphinxhyphen{}Access:} The access\sphinxhyphen{}edge will need to support multiple
access technologies (e.g., WiFi, 5G, fiber), and allow users to
seamlessly move between them. Research is needed to break down
existing technology silos, and design converged solutions to common
problems (e.g., security, mobility, QoS).

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Heterogeneity:} Since the access\sphinxhyphen{}edge will be about low\sphinxhyphen{}latency
and high\sphinxhyphen{}bandwidth connectivity, much edge functionality will be
implemented by programming the forwarding pipeline in white\sphinxhyphen{}box
switches, and more generally, will use other domain\sphinxhyphen{}specific
processors (e.g., GPUs, TPUs). Research is needed to tailor edge
services to take advantage of heterogeneous resources, as well as
how to construct end\sphinxhyphen{}to\sphinxhyphen{}end applications from such a collection of
building blocks.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Virtualization:} The access\sphinxhyphen{}edge will virtualize the underlying
hardware using a range of techniques, from VMs to containers to
serverless functions, interconnected by a range of L2, L3, and L4/7 virtual
networks, some of which will be managed by SDN control
applications. Research is needed to reconcile the assumptions made
about by cloud native services and access\sphinxhyphen{}oriented Virtualized
Network Functions (VNFs) about how to virtualize compute, storage,
and networking resources.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Multi\sphinxhyphen{}Tenancy:} The access\sphinxhyphen{}edge will be multi\sphinxhyphen{}tenant, with
potentially different stakeholders (operators, service providers,
application developers, enterprises) responsible for managing
different components. It will not be feasible to run the entire
access\sphinxhyphen{}edge in a single trust domain, as different components will
operate with different levels of autonomy. Research is needed to
minimize the overhead isolation imposes on tenants.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Customization:} Monetizing the access\sphinxhyphen{}edge will require the
ability to offer differentiated and customized services to different
classes of subscribers/applications. Sometimes called network
slicing (see Section 5.3), this involves support for performance isolation at the
granularity of service chains—the sequence of functional elements
running on behalf of some subscriber. Research is needed to enforce
performance isolation in support of service guarantees.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Near\sphinxhyphen{}Real Time:} The access\sphinxhyphen{}edge will be a highly dynamic
environment, with functionality constantly adapting in response to
mobility, workload, and application requirements. Supporting such an
environment requires tight control loops, with control software
running at the edge. Research is needed to analyze control loops,
define analytic\sphinxhyphen{}based controllers, and design dynamically adaptable
mechanisms.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Data Reduction:} The access\sphinxhyphen{}edge will connect an increasing
number of devices (not just humans and their handsets), all of which
are capable of generating data. Supporting data reduction will be
critical, which implies the need for substantial compute capacity
(likely including domain\sphinxhyphen{}specific processors) to be available in the
access\sphinxhyphen{}edge. Research is needed to refactor applications into their
edge\sphinxhyphen{}reduction/backend\sphinxhyphen{}analysis subcomponents.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Distributed Services:} Services will become inherently
distributed, with some aspects running at the access\sphinxhyphen{}edge, some
aspects running in the datacenter, and some running on\sphinxhyphen{}premises or
in an end device (e.g., on\sphinxhyphen{}vehicle). Supporting such an environment
requires a multi\sphinxhyphen{}cloud solution that is decoupled from any single
infrastructure\sphinxhyphen{}based platform, with research needed to develop
heuristics for function placement.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Scalability:} The access\sphinxhyphen{}edge will potentially span thousands or
even tens of thousands of edge sites. Scaling up the ability to
remotely orchestrate that many edge sites (even at just the
infrastructure level) will be a qualitatively different challenge
than managing a single datacenter. Research is needed to scale both
the edge platform and widely deployed edge services.

\end{itemize}
\phantomsection\label{\detokenize{cloud:reading-democratizing}}
\begin{sphinxadmonition}{note}{Further Reading}

\sphinxAtStartPar
To better understand the research opportunity at the access\sphinxhyphen{}edge,
see \sphinxhref{https://ccronline.sigcomm.org/wp-content/uploads/2019/05/acmdl19-289.pdf}{Democratizing the Network Edge}.
ACM SIGCOMM CCR, April 2019.
\end{sphinxadmonition}


\chapter{About The Book}
\label{\detokenize{README:about-the-book}}\label{\detokenize{README::doc}}
\sphinxAtStartPar
Source for \sphinxstyleemphasis{5G Mobile Networks: A Systems Approach} is available on
GitHub under
terms of the \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{Creative Commons (CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0)} license. The
community is invited to contribute corrections, improvements, updates,
and new material under the same terms. While this license does not
automatically grant the right to make derivative works, we are keen to
discuss derivative works (such as translations) with interested
parties. Please reach out to \sphinxhref{mailto:discuss@systemsapproach.org}{discuss@systemsapproach.org}.

\sphinxAtStartPar
If you make use of this work, the attribution should include the
following information:

\begin{DUlineblock}{0em}
\item[] \sphinxstyleemphasis{Title: 5G Mobile Networks: A Systems Approach}
\item[] \sphinxstyleemphasis{Authors: Larry Peterson and Oguz Sunay}
\item[] \sphinxstyleemphasis{Source:} \sphinxurl{https://github.com/SystemsApproach/5G}
\item[] \sphinxstyleemphasis{License:} \sphinxhref{https://creativecommons.org/licenses/by-nc-nd/4.0}{CC BY\sphinxhyphen{}NC\sphinxhyphen{}ND 4.0}
\end{DUlineblock}


\section{Read the Book}
\label{\detokenize{README:read-the-book}}
\sphinxAtStartPar
This book is part of the \sphinxhref{https://www.systemsapproach.org}{Systems Approach Series}, with an online version published at
\sphinxhref{https://5g.systemsapproach.org}{https://5G.systemsapproach.org}.

\sphinxAtStartPar
To track progress and receive notices about new versions, you can follow
the project on
\sphinxhref{https://www.facebook.com/Computer-Networks-A-Systems-Approach-110933578952503/}{Facebook}
and \sphinxhref{https://twitter.com/SystemsAppr}{Twitter}. To read a running
commentary on how the Internet is evolving, follow the \sphinxhref{https://systemsapproach.substack.com}{Systems Approach
on Substack}.


\section{Build the Book}
\label{\detokenize{README:build-the-book}}
\sphinxAtStartPar
To build a web\sphinxhyphen{}viewable version, you first need to download the source:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYGZdl{} mkdir \PYGZti{}/systemsapproach 
\PYGZdl{} cd \PYGZti{}/systemsapproach 
\PYGZdl{} git clone https://github.com/SystemsApproach/5G.git 
\PYGZdl{} cd 5G
\end{sphinxVerbatim}

\sphinxAtStartPar
The build process is stored in the Makefile and requires Python be
installed. The Makefile will create a virtualenv (\sphinxcode{\sphinxupquote{venv\sphinxhyphen{}docs}}) which
installs the documentation generation toolset. You may also need to
install the \sphinxcode{\sphinxupquote{enchant}} C library using your system’s package manager
for the spelling checker to function properly.

\sphinxAtStartPar
To generate HTML in \sphinxcode{\sphinxupquote{\_build/html}},  run \sphinxcode{\sphinxupquote{make html}}.

\sphinxAtStartPar
To check the formatting of the book, run \sphinxcode{\sphinxupquote{make lint}}.

\sphinxAtStartPar
To check spelling, run \sphinxcode{\sphinxupquote{make spelling}}. If there are additional
words, names, or acronyms that are correctly spelled but not in the dictionary,
please add them to the \sphinxcode{\sphinxupquote{dict.txt}} file.

\sphinxAtStartPar
To see the other available output formats, run \sphinxcode{\sphinxupquote{make}}.


\section{Contribute to the Book}
\label{\detokenize{README:contribute-to-the-book}}
\sphinxAtStartPar
We hope that if you use this material, you are also willing to
contribute back to it. If you are new to open source, you might check
out this \sphinxhref{https://opensource.guide/how-to-contribute/}{How to Contribute to Open
Source} guide. Among
other things, you’ll learn about posting \sphinxstyleemphasis{Issues} that you’d like to see
addressed, and issuing \sphinxstyleemphasis{Pull Requests} to merge your improvements back
into GitHub.

\sphinxAtStartPar
If you’d like to contribute and are looking for something that needs
attention, see the \sphinxhref{https://github.com/SystemsApproach/5G/wiki}{wiki}
for the current TODO list.


\chapter{About The Authors}
\label{\detokenize{authors:about-the-authors}}\label{\detokenize{authors::doc}}
\sphinxAtStartPar
\sphinxstylestrong{Larry Peterson} is the Robert E. Kahn Professor of Computer
Science, Emeritus at Princeton University, where he served as Chair
from 2003\sphinxhyphen{}2009. He is a co\sphinxhyphen{}author of the best selling networking
textbook \sphinxstyleemphasis{Computer Networks: A Systems Approach (6e)}, which is now
available as open source on GitHub. His research focuses on the
design, implementation, and operation of Internet\sphinxhyphen{}scale distributed
systems, including the widely used PlanetLab and MeasurementLab
platforms.  He is currently leading the CORD and Aether access\sphinxhyphen{}edge
cloud projects at the Open Networking Foundation (ONF), where he
serves CTO.  Professor Peterson is a member of the National Academy of
Engineering, a Fellow of the ACM and the IEEE, the 2010 recipient of
the IEEE Kobayashi Computer and Communication Award, and the 2013
recipient of the ACM SIGCOMM Award. He received his Ph.D. degree from
Purdue University in 1985.

\sphinxAtStartPar
\sphinxstylestrong{Oguz Sunay} is currently the Vice President for Research \&
Development at ONF, where he leads all mobile\sphinxhyphen{}related projects. Prior
to that, he served as the Chief Architect for Mobile Networking at
ONF. Before joining ONF, Sunay was the CTO at Argela\sphinxhyphen{}USA, where he was
the innovator of a Programmable Radio Access Network Architecture
(ProgRAN) for 5G that enabled the world’s first dynamically
programmable RAN slicing solution. He has also held prior industry
positions at Nokia Research Center and Bell Laboratories, where he
focused on 3G and 4G end\sphinxhyphen{}to\sphinxhyphen{}end systems architectures and participated
and chaired various standardization activities. Sunay has also spent
over 10 years in academia, as a Professor of Electrical and Computer
Engineering. He holds many U.S. and European patents on various
aspects of 3G, 4G, and 5G, and has authored numerous journal and
conference publications. He received his Ph.D. and M.Sc. from Queen’s
University, Canada, and his B.Sc.Hon. from METU, Turkey.


\chapter{Read the Latest!}
\label{\detokenize{latest:read-the-latest}}\label{\detokenize{latest::doc}}
\sphinxAtStartPar
\sphinxhref{https://systemsapproach.substack.com/}{Substack Newsletter:} Stay
up\sphinxhyphen{}to\sphinxhyphen{}date with the latest developments by following the authors on
\sphinxhref{https://systemsapproach.substack.com/}{Substack}, where they
connect the concepts and lessons in this book to what’s happening in
the Internet today.

\sphinxAtStartPar
\sphinxhref{https://systemsapproach.org/books/}{Book Series:} Also check out
our companion books that cover emerging topics in more depth.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://sdn.systemsapproach.org}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://tcpcc.systemsapproach.org}{TCP Congestion Control: A Systems Approach}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://ops.systemsapproach.org}{Edge Cloud Operations: A Systems Approach}

\end{itemize}


\chapter{Print Copies}
\label{\detokenize{print:print-copies}}\label{\detokenize{print::doc}}
\sphinxAtStartPar
We make all books in the \sphinxstyleemphasis{Systems Approach} series available as both
print and e\sphinxhyphen{}books. This book is available via Amazon: \sphinxhref{https://amzn.to/3EjwMH0}{5G Mobile Networks: A Systems Approach}

\sphinxAtStartPar
\sphinxhref{https://systemsapproach.org/books/}{Book Series:} Also check out
our companion books that cover networking and emerging topics in more depth.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://amzn.to/3CtG81U}{Computer Networks: A Systems Approach}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://amzn.to/3rmLdCP}{Software\sphinxhyphen{}Defined Networks: A Systems Approach}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://amzn.to/3UTYi3T}{TCP Congestion Control: A Systems Approach}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://amzn.to/3MfvK13}{Edge Cloud Operations: A Systems Approach}

\end{itemize}

\sphinxAtStartPar
As participants in the Amazon Associate program we may earn income from qualifying purchases using the links above.



\renewcommand{\indexname}{Index}
\printindex
\end{document}