\section{Introduction}\label{sec:intro}

{This paper is motivated by} the fact that fluctuations in the
demand {for electricity} to be met by conventional thermal
generating units typically result in significantly increased, and
nontrivial,  ancillary costs. Today, such demand fluctuations are
mainly due to time-dependent
  consumer preferences. In addition, in the future, a certain percentage of electricity
production is required by law in many states in the U.S. to come
from renewable resources \citep{BW08}. The dramatic
 volatility  of renewable energy
 resources may aggravate the variability of the demand for conventional thermal
generators and result in significant ancillary cost.
 More concretely, either a demand surge or a decrease in renewable generation may result in
(i) higher energy costs due to the deployment of peaking plants with higher ramping rates but higher marginal cost, such as
 oil/gas combustion turbines, and (ii) the redispatch cost\footnote{A
certain level of reserve must always be maintained in an electric
power system. Local reserve shortages are usually due to the quick
increase of system load rather than a capacity deficiency. If the
increase of system load makes the system short in reserves, the
system will redispatch resources to increase the amount of reserves
available. Redispatch generally increases the generation cost
and results in higher prices. The redispatch cost
can be very high (cf. Section 2.3.2 of \cite{E10}).} that the system will incur to meet reserve
constraints if the increase of demand (or decrease of renewable generation) causes a reserve shortage.

There is general agreement that
  charging real-time prices (that reflect current operating conditions)
 to electricity consumers has the potential of reducing supplier ancillary cost, improving
 system efficiency, and lowering volatility in wholesale prices \citep{DOE06,SL08,C10}.
Therefore, dynamic pricing, especially real-time marginal cost
pricing, is often identified as a priority for the implementation of
 wholesale electricity markets with responsive demand \citep{H10},
 which in turn raises many new questions. For example,
 should prices for a given time interval be calculated ex ante or ex post? Does real-time pricing introduce
 the potential for new types of market instabilities? How is supplier competition affected? In this paper, we abstract away
 from {almost all of these} questions and focus on the specific issue of whether prices should also explicitly encourage consumers to adapt their
demands so as to reduce supplier ancillary cost.

To illustrate the issue that we focus on, we note that a basic model
of electricity markets assumes that the cost of satisfying a given
level $A_t$ of aggregate demand during period $t$ is of the form
$C(A_t)$. It then follows that in a well-functioning wholesale
market, the observed price should more or less reflect the marginal
cost $C'(A_t)$. {In particular,} prices should be more or less
determined by the aggregate demand level. Empirical data do not
quite support this view. Fig. \ref{Fig:211} plots the real-time
system load and the hourly prices on February 11, 2011 and on
February 16, 2011, as reported by the New England ISO \citep{F11}.
We observe that prices do not seem to be determined solely by $A_t$
{but that the} changes in demand, $A_t-A_{t-1}$, {also} play a major
role. In particular, the largest prices seem to occur after a demand
surge, and not necessarily at the hour when the load is highest.
We take this as evidence that the total cost over $T+1$ periods is
not of the form
$$
\sum_{t=0}^T C(A_t),
$$
but rather of the form
\begin{equation}\label{eq:aa}
\sum_{t=0}^T \big(C(A_t)+H(A_{t-1},A_{t})\big),
\end{equation}
for a suitable function $H$.

\begin{figure}
\begin{minipage}[t]{0.5\linewidth}
\includegraphics[width=3.02in]{211.pdf}
\end{minipage}%
\begin{minipage}[t]{0.5\linewidth}
\centering
\includegraphics[width=3.01in]{0216.pdf}
\end{minipage}
\caption{Real-time prices and actual system load, ISO New England
Inc. Blue bars represent the real-time system loads and the dots
connected by a black line represent the hourly
prices.}\label{Fig:211}
\end{figure}

We take the form of Eq.\ (\ref{eq:aa}) as our starting point and raise the question of the appropriate prices.
A naive view would argue that at time $t$, $A_{t-1}$ has already been realized, and taking its value for granted,
a consumer should be charged a unit price equal to
\begin{equation}\label{eq:two}
C'(A_t) +\frac{\partial}{\partial A_t} H(A_{t-1},A_t),
\end{equation}
which is the supplier's marginal cost at stage $t$. We refer to this
naive approach as ``marginal cost pricing'' (MCP). However, a simple
argument based on standard mathematical programming optimality
conditions shows that for system optimality to obtain, the demand
$A_{t-1}$ should also incur
 (after $A_t$ is realized) a unit price of
\begin{equation}\label{eq:three}
\frac{\partial}{\partial A_{t-1}} H(A_{t-1},A_t).
\end{equation}
In day-ahead markets, suppliers typically carry out an intertemporal
optimization, and it is reasonable to expect that the two types of
marginal costs, captured by Eqs.\ \eqref{eq:two} and
\eqref{eq:three}, are both properly accounted for. However, in
current real-time balancing markets, once $A_{t-1}$ is realized, a
supplier will aim at charging the marginal cost in Eq.\
\eqref{eq:two}, but will be unable to charge the additional marginal
cost in Eq.\ \eqref{eq:three} to the past demand $A_{t-1}$. In
contrast, the pricing mechanism that we propose and analyze in this
paper is designed to include the additional marginal cost in Eq.
\eqref{eq:three}.\footnote{In current two-settlement systems, the
real-time prices are charged only on the difference of the actual
demand and the estimated demand at the day-ahead market. However,
the two-settlement  system provides the same real-time incentives to
price-taking consumers, as if they were purchasing all of their
electricity at the real-time prices (cf. Chapter 3-2 of
\cite{S02}).}

The actual model that we consider will be richer from the one discussed above
 in a number of respects. It includes an exogenous source of uncertainty (e.g., {representing}
 weather conditions) that has an impact on consumer utility
  and supplier cost, {and therefore the model can incorporate the effects of volatile} renewable electricity production.
    It  allows for consumers with internal state variables (e.g., a consumer's
  demand may be affected by how much electricity she has already used). It also allows for multiple consumer types
  (i.e., with different utility functions and different internal state dynamics). Consumers are generally modeled as price-takers,
  as would be the case in a model involving an infinity (a continuum) of consumers. However, we also consider the case of finite
  consumer populations and explore certain equilibrium concepts that are well-suited to the case of finite but large consumer populations.
  On the other hand, we ignore most of the distinctions between ex post and ex ante prices. Instead, we assume that at each time step,
  the electricity market clears. The details of how this could happen are important, but are generic to electricity markets,
  hence not specific to our models, and somewhat
  orthogonal to the subject of this paper. (See however Appendix \ref{sec:im} for some discussion of implementation issues.)

The ancillary cost function $H(A_{t-1},A_t)$ is of course a central
element of our model. How can we be sure that this is the right
form? In general, redispatch and reserve dynamics are complicated
{and} one should not expect such a function to capture all of the
complexity of the true system costs; perhaps, a more complex
functional form such as $H(A_{t-2},A_{t-1},A_t)$ would be more
appropriate. We believe that the form we have chosen is a good
enough  approximation, at least under certain conditions. To argue
this point, we present in Appendix \ref{sec:A0} an example that
involves a more detailed system model (in which the true cost is a
complicated function of the entire history of demands) and show that
a function of the form $H(A_{t-1},A_t)$
 can capture most of the cost of ancillary services.

\subsection{Summary of contributions}
Before continuing, we provide here a roadmap of the paper together
with a summary of our main contributions.
\begin{itemize}{\itemsep=0pt}
\item[(a)]
We provide a stylized (yet quite rich) model of an electricity
market, which  incorporates the cost of ancillary services (cf. Section \ref{sec:model}).
\item[(b)] We provide some justification of the form of the cost function in our model, as a reasonable approximation of more detailed physical models
(cf. Appendix \ref{sec:A0}).
\item[(c)] We propose a pricing mechanism that properly charges for the effects of consumer actions on  ancillary services (cf. Section \ref{sec:PM}).
\item [(d)] For a continuum model involving nonatomic price-taking consumers, we consider  Dynamic Oblivious Equilibria (DOE),
in which every consumer maximizes her expected payoff under the
sequence of prices induced by {a} DOE strategy profile (Section
\ref{sec:DOE}). We show that (under standard convexity assumptions),
our mechanism maximizes social welfare (cf. Theorem \ref{thm:SO_social} {in Section \ref{sec:social}}).

\item[(e)] We {carry out a game-theoretic analysis of} the case of a large but finite number of consumers.
 We show that a large population of consumers who act
according to a DOE (derived from an associated continuum game)
 results in asymptotically optimal (as the number of consumers goes to infinity) social welfare (cf. Theorem \ref{thm:SO_social}
in Section \ref{sec:social}), {and asymptotically maximizes every
consumer's expected payoff} (this is an ``asymptotic Markov
equilibrium'' property; cf.\ Theorem \ref{thm:APE} {in Section
\ref{sec:AMP}}).

\item[(f)] We illustrate the potential benefits of our mechanism through a simple numerical example. In particular, we show that compared with
marginal cost pricing, the proposed mechanism reduces the peak load,
and therefore has the potential to reduce the need for long-term
investments in peaking plants (cf.\ Appendix \ref{sec:num}).

\end{itemize}

\subsection{Related literature}

There are two streams of literature, {on electricity pricing and on
game theory, that are relevant to our work, and which we now proceed
to discuss, while also highlighting the differences from the present
work.}

Regarding electricity markets, the impact of supply friction on
economic efficiency and price volatility has received some recent
attention.  \cite{M08} shows that under ramping constraints, the
prices faced by consumers may not necessarily equal the true
supplier marginal cost. In a continuous-time competitive market
model, \cite{CM10} show that the limited capability of generating
units to meet real-time demand, due to
 relatively low ramping rates, does not harm social welfare,
but may result in extreme price fluctuations. In a similar spirit,
\cite{KM10} construct a dynamic game-theoretic model to study the
tradeoff between economic efficiency and price volatility. Closer to
the present paper, \cite{CM10b} construct a dynamic newsboy model to
study the reserve management problem in electricity markets, where
the demand is assumed to be exogenous. The supplier cost in their
model depends not only on the overall demand, but also on the
generation resources used to satisfy the demand. For example, a
quickly increasing demand may require more responsive and more
expensive resources (e.g., peaking generation plants).

{To study the impact of pricing mechanisms on consumer behavior and
load fluctuations, we construct a dynamic game-theoretic model that
differs from existing dynamic models for electricity markets and
incorporates both the consumers' responses to real-time price
fluctuations and the suppliers' ancillary cost incurred by load
swings.}
Some major differences between our model and existing ones are discussed
at the end of Section \ref{sec:model}.

On the game-theoretic side,  {the standard solution concept for
stochastic dynamic  games is {the} \textbf{Markov perfect
equilibrium} (MPE) \citep{FT91,MT88}, {involving strategies} where
an agent's action depends on the current state of all agents. As the
number of agents grows large, the computation of an MPE is often
intractable \citep{DP07}. For this reason, alternative equilibrium
concepts, for related games featuring a nonatomic continuum of
agents (e.g., ``oblivious equilibrium'' or ``stationary
equilibrium'' for dynamic games without aggregate shocks), have
received much recent attention \citep{WBV08,AJ10}.}

There is a large literature on a variety of approximation properties
of nonatomic equilibria \citep{MV93,A04,A08}. Recently,
\cite{AJ10} derive
 sufficient conditions for a
stationary equilibrium strategy to have the \textbf{Asymptotic
Markov Equilibrium} (AME) property, i.e., a stationary
equilibrium strategy asymptotically maximizes every agent's expected
payoff (given that all the other agents use the same stationary
equilibrium  strategy), as the number of agents grows large. {Their
model includes random shocks that} are assumed to be idiosyncratic
across agents. However, in the problem that we are interested in, it
is important to incorporate aggregate shocks (such as weather
conditions) that have a global impact on all agents. In this spirit,
\cite{WBV10} consider a market model with aggregate profit shocks,
and study an equilibrium concept at which every firm's strategy
depends on the firm's current state and on the recent history of the
aggregate shock. For a general dynamic game model with aggregate
shocks, \cite{BC10} shows that a nonatomic {counterpart} of an MPE,
which we refer to as a \textbf{Dynamic Oblivious Equilibrium (DOE)}
in this paper, asymptotically approximates an MPE in the sense that
as the number of agents increases to infinity, the actions taken in
an MPE can be well approximated by those taken by a DOE strategy of
the nonatomic limit game. However, without further restrictive
assumptions on the agents' state transition kernel, the
approximation property of the actions taken by a DOE strategy
 does not imply the AME property of the DOE, and we are not aware of
 any AME results for models that include aggregate shocks. Our work is different in this respect:
for a dynamic nonatomic model with aggregate shocks, which
is a simplified variation of the general model considered in
\cite{BC10}, we prove the AME property of a DOE.

The efficiency of nonatomic equilibria for static games has been
addressed in recent research \citep{RT04,M04,BC11}. For a dynamic
industry model with a continuum of identical producers and exogenous
aggregate shocks, \cite{LP71} show (under convexity assumptions)
that the expected social welfare is maximized at a unique
competitive equilibrium. In a similar spirit, in this paper we show
(under convexity assumptions) that the proposed pricing mechanism
maximizes the expected social welfare in a model involving a
continuum of (possibly heterogeneous) consumers. We also consider
the case of a large but finite number of consumers, and show that
the expected social welfare is approximately maximized if all
consumers act according to a nonatomic equilibrium (DOE). For large
dynamic games, the asymptotic social optimality of nonatomic
equilibria (DOEs) established
 in this paper seems to be new.

\old{
\subsection{Outline of the paper}

The rest of the paper is organized as follows. In Section
\ref{sec:model}, we  introduce the dynamic game model. In Section
\ref{sec:PM}, inspired by the observation that the marginal cost
pricing may not be socially optimal in a market with friction (see
Example \ref{ex:price}), we propose an alternative dynamic pricing
mechanism. In Section \ref{sec:DOE}, we give the formal definition
of a DOE. In Section \ref{sec:AMP}, we show that as the number of
consumers increases to infinity, every consumer's expected payoff is
approximately maximized by a DOE strategy, if the other consumers
follow the DOE strategy. In Section \ref{sec:social}, we show that
under the proposed pricing mechanism, social welfare is maximized at
a DOE of a continuum game. Also, for a dynamic game with finitely
many consumers, and under a certain convexity assumption, we
validate the asymptotic social optimality of the proposed pricing
mechanism: as the number of consumers grows large, if all consumers
use a DOE strategy, then social welfare is approximately maximized.
In Section \ref{sec:num}, we present several numerical examples to
compare the proposed pricing mechanism with marginal cost pricing.
Finally, in Section \ref{sec:con}, we make some brief concluding
remarks and discuss some directions for future work.}

\section{Model}\label{sec:model}

We consider a $(T+1)$-stage dynamic game with the following
elements:

\begin{enumerate}{\itemsep=0pt}

\item The game is played in \textbf{discrete time}. We index the time
periods with
 $t=0,1,\ldots,T$. Each stage may represent a five minute interval in real-time balancing markets
 where prices and dispatch solutions are typically provided at five minute intervals.

\item There are $n$ \textbf{consumers}, indexed by $1,\ldots,n$.

\item At each stage $t$, let $s_t \in \mathcal {S} $ be an \textbf{exogenous
state}, which evolves as a Markov chain and whose transitions are
{not affected by} consumer actions. The set $\mathcal {S} $ is
assumed to be finite. In electricity markets, the exogenous  state
may represent time and/or weather conditions, which impact
consumer utility and supplier cost. It may also represent the level
of renewable generation.

\item  For notational conciseness, for $t \ge 1$, let
$\overline s_t=(s_{t-1},s_t)$, and let $\overline s_0=s_0$. We use
$\overline {\mathcal {S}}_t$ to denote the set of all possible
$\overline s_t$.  We  refer to $\overline s_t$ as the \textbf{global
state} at stage $t$.

\item Given an initial global state $s_0$,
the \textbf{initial states (types) of the consumers},
$\{x_{i,0}\}_{i=1}^n$,
 are independently drawn according to a
probability measure $\eta_{s_0}$ over a finite set $\mathcal {X}_0$.
We use $X$ to denote the cardinality of $\mathcal {X}_0$.

\item
The \textbf{state of consumer} $i$ at stage $t$ is denoted by
$x_{i,t}$. At $t=0$, consumer $i$'s initial state, $x_{i,0}$,
indicates her type. For $t=1,\ldots,T$, we have
$x_{i,t}=(x_{i,0},z_{i,t})$, where $z_{i,t} \in \mathcal {Z}$ and
$\mathcal {Z}=[0,Z]$ is a compact subset of $\mathbb{R}$. The
variables $\{z_{i,t}\}_{i=1}^n$ allow us to model  intertemporal
substitution effects
 in consumer $i$'s demand.

\item We use $\mathcal {X}_t$ to denote a \textbf{consumer's state space} at
stage $t$. In particular, at stage $t \ge 1$, $\mathcal
{X}_t=\mathcal {X}_0 \times \mathcal {Z}$.

\item At stage $t$,  consumer $i$ takes an \textbf{action} $a_{i,t}$
and receives a nonnegative \textbf{utility}\footnote{At $t = 0$,
$U_0$ is a mapping  from $\mathcal {X}_0  \times \mathcal {S} \times
\mathcal {A}$ to $[0,\infty) $, while for $t \ge 1$, $U_t$ is a
mapping from $\mathcal {X}_0 \times {\mathcal {Z}} \times \mathcal
{S} \times \mathcal {A}$ to $[0,\infty) $.}
$U_t(x_{i,t},s_t,a_{i,t})$.

\item Each consumer's \textbf{action space} is
$\mathcal {A}=[0,B]$, where $B$ is a positive real number. (In the electric power context, $B$ could reflect
 a local transmission capacity constraint.)

\item  We
 use $A_t=\sum\nolimits_{i=1}^n a_{i,t}$ to denote the \textbf{aggregate
 demand} at stage $t$.

\item Given consumer $i$'s
 current state, $x_{i,t}$, and the next exogenous state $s_{t+1}$,
the
 next state of consumer $i$ is determined by
her action
 taken at stage $t$, i.e., $x_{i,t+1}=\left( x_{i,0},
z_{i,t+1}\right)$, where $z_{i,t+1}=
 r(x_{i,t},a_{i,t},s_{t+1})$, for a given function $r$.

\item Let $G_t=A_t+R_t$ be the \textbf{capacity} available at stage $t$,
where $R_t$ is the system \textbf{reserve} at stage $t$. For
simplicity, we assume that the system reserve at stage $t$ depends
only on the current aggregate demand, $A_t$, and the current
exogenous state $s_t$. That is, we have $R_t= g(A_t,s_t)$ for a
given function of $g$ that reflects the reserve policy of the system
operator.

\item At stage $t$, let $\overline C(A_t, R_t, s_t)$ {be the total conventional \textbf{generation cost}, that is, the sum}
of the supplier's cost to meet the aggregate
demand $A_t$ through its primary energy resources, e.g., base-load
power plants, and the cost to maintain a system reserve $R_t$.
 Since $R_t$ depends only on $A_t$ and $s_t$, we can write
$\overline C(A_t, R_t, s_t)$ as a function of $A_t$ and $s_t$, i.e.,
there exists a \textbf{primary cost function} $C: \mathbb{R} \times
\mathcal {S} \to [0,\infty)$ such that $C(A_t,s_t)=\overline C(A_t,
R_t, s_t)$. We assume that for any $s \in \mathcal {S}$,
$C(\cdot,s)$ is nondecreasing.

\item  At stage $t \ge 1$, let $\overline H(A_{t-1},A_{t},R_{t-1},R_{t},s_t)$ denote
the ancillary
 cost incurred by load swings\footnote{In general, the supplier
ancillary cost may depend on the entire history of system load and
global states. However, {ancillary cost functions with the simple
form} $\overline H(A_{t-1},A_{t},R_{t-1},R_{t},s_t)$ can serve as a
good approximation of the supplier's true ancillary cost (cf.
Appendix \ref{sec:A0}). }. Since $R_t$ depends only on $A_t$ and
$s_t$, we can write $\overline H(A_{t-1},A_{t},R_{t-1},R_{t},s_t)$
as a function of $A_{t-1}$, $A_t$, $s_{t-1}$, and $s_t$, i.e., there
exists an \textbf{ancillary cost function} $H: \mathbb{R}^2 \times
\mathcal {S}^2 \to [0,\infty)$ such that $\overline H = H(A_{t-1},
A_t, \overline s_t)$. The ancillary cost at stage $0$ is assumed to
be a function of $s_0$ and $A_0$.

\item At stage $0$,  the total supplier cost is of the form
\begin{equation}\label{equa:cost0}
C(A_0,s_0)+  H_0(A_{0}, s_0),
\end{equation}
and for  $t=1,\ldots,T$,
 the total supplier cost at
stage $t$ is given by
\begin{equation}\label{equa:cost}
C(A_t,s_t)+   H(A_{t-1}, A_t, \overline s_t).
\end{equation}

\end{enumerate}

In contrast to existing dynamic models for electricity markets with
an exogenous demand process \citep{CM10,CM10b}, our dynamic
game-theoretic model incorporates the consumer reactions to price
fluctuations, and allows us to study the impact of pricing
mechanisms on consumer behavior and economic efficiency. Through a
dynamic game-theoretic formulation, \cite{KM10} study the tradeoff
between economic efficiency and price volatility. Our model is
different from the one studied in \cite{KM10} in the following
{respects:}

\begin{enumerate}{\itemsep=0pt}
\item
 Our model allows the generation cost to
depend on  an exogenous state, and therefore {can incorporate}
supply-side volatility due to  uncertainty in renewable electricity
generation. As an example, consider a case where the exogenous
state, $s_t$, represents the electricity generation from renewable
resources at stage $t$. Then the demand for conventional generation
is $A_t-s_t$. Suppose that the system reserve is proportional to the
system load, say, $\delta A_t$ {for some constant} $\delta>0$. The
cost function, $\overline C(A_t,R_t,s_t)$, then depends only on the
output of conventional generating units, $A_t-s_t$, and the system
reserve, $\delta A_t$. The ancillary cost occurred at stage $t$
depends on the system reserve and the outputs of conventional
generating units at stages $t-1$ and $t$, and is therefore a
function of $A_{t-1}$, $A_t$, $s_{t-1}$, and $s_t$.

\item More important, instead of penalizing
each consumer's attempt to change her {own} action across time,
 the ancillary cost function in our model penalizes the change in
 the aggregate demand by all consumers.  The change in a single
 consumer's action may harm or benefit the social welfare, while the
 volatility of the aggregate demand is usually undesirable.
\end{enumerate}

The main feature of our model is the ancillary cost function $H$,
which makes the supplier cost nonseparable over time. In an electric
power system, the ancillary cost function models the costs
associated with the variability of conventional thermal generator
output, such as the energy cost of peaking plants and the redispatch
cost. Note that the ancillary cost is not necessarily zero when $A_t
\le A_{t-1}$, because thermal generating units have ramping-down
constraints, and because a decrease in renewable electricity
production may {lead to an increase of} the system reserve, even if
$A_t \le A_{t-1}$.
 The presence of the ancillary cost function makes {conventional}
 marginal cost pricing  inefficient (cf. Example \ref{ex:price} in Section \ref{sec:PM}).

\old{ For the energy conservation problem in data centers, A similar
model is used to study the energy consumption in data centers in
\cite{LW11}, where the switching cost}

To keep the model simple, we do not incorporate any idiosyncratic
randomness in  consumer state evolution. Thus, besides the
randomness of consumer types (initial states),
 the only source of stochasticity in the model {is}
the exogenous state $s_t$.

To effectively highlight the impact of pricing mechanisms on
consumer behavior, as well as on economic efficiency and demand
volatility, we have made the following simplifications and
assumptions for the power grid:

\begin{itemize}
\item [(a)]
 As in \cite{CM10}, we assume that the physical production capacity is  large enough so that the possible
 changes
of the generation capacity are not constrained.

\item [(b)]  Transmission capacity is large enough to avoid any
congestion. We also assume that the cost of supplying electricity to
consumers {at} different locations is the same. Therefore, a common
price for all consumers is appropriate.

\item [(c)]  We use a simplified form of ancillary cost functions,
$\overline H(A_{t-1},A_{t},R_{t-1},R_{t},s_t)$, to approximate the
supplier ancillary cost. \old{ Such an
approximation is well justified if the ancillary energy service,
e.g., a peaking plant, is deployed only at a single stage. At
the next stage, either the demand drops, or the primary
energy resource can ramp up rapidly enough to meet the increasing
demand. This is usually the case, even in current electricity
markets where most consumers pay a fixed price for the electricity
they consume. For example, among the $239$ deployments of New York
independent system operator reserve in $2002$, most of them were
$12$ minutes or less in duration \cite{K03}. Under real-time
pricing, it may not be implausible to suppose that this
approximation will be even better justified with the price-responsive
demand.}In Appendix \ref{sec:A0}, we present a numerical example to
justify this approximation.

\old{ Numerical results show that the difference approximation

In general, the supplier ancillary cost may depend on the entire
history of system load and exogenous states. We assume that the system
reserve is a function of the current aggregate demand and the
current exogenous state. Under this assumption,

This assumption allows us to consider a simplified form of ancillary
cost functions, which depends only on $A_{t-1}$, $A_t$ and
$\overline s_t$.}
\end{itemize}

\section{The Pricing Mechanism}\label{sec:PM}

The marginal cost pricing mechanism discussed in Section \ref{sec:intro}
charges a time-varying unit
price on each consumer's demand.
As demonstrated in the following example, a time-varying price that equals the supplier's instantaneous
  marginal cost may not achieve social optimality in a {setting that includes ancillary costs.} For this reason, we
propose a new pricing mechanism that takes into account the
ancillary cost associated with a consumer's demand at the {{\bf
previous}} stage.

\begin{example}\label{ex:price}  \rm{
Consider a two-stage deterministic model with one
 consumer and one supplier. At stage $t$, the consumer's utility function
is $U_t:[0,\infty) \to [0,\infty)$.
  Let
$a_t$ denote the demand at stage $t$, and let $\v{a}=(a_0,a_1)$.
  Let
$g_t$ denote the actual generation at stage $t$, and let
$\v{g}=(g_0,g_1)$.  Two unit prices, $p_0$ and $p_1$, are charged on
the consumption at stage $0$ and $1$, respectively. Let
$\v{p}=(p_0,p_1)$.  The consumer's payoff-maximization problem is
\begin{equation}\label{eq:exa_C}
 { \mathop {\rm Maximize}\limits_{\v{a}}}\;\;\;\;\;
U_0(a_0)- p_0 a_0 + U_1(a_1)- p_1 a_1.
\end{equation}
Let $H_0$ be identically zero, and let the ancillary cost function
at stage $1$ depend only on the difference between the supply at the
two stages. That is, the ancillary cost at stage $1$ is of the form
$H(g_1-g_0)$. The supplier's profit-maximization problem is
\begin{equation}\label{eq:exa_S}
\displaystyle { \mathop {\rm Maximize}\limits_{\v{g}}}\;\;\;\;\; p_0
g_0 + p_1 g_1-C(g_0)-C(g_1)-H(g_1-g_0).
\end{equation}
The social planner's problem is
\begin{equation}\label{eq:exa_W}
\begin{array}{l}
\displaystyle {\mathop {\rm Maximize}\limits_{(\v{a},\v{g})}}\;\;\;\;\; U_0(a_0)+
U_1(a_1)-C(g_0)-C(g_1)-H(g_1-g_0) \\[5pt]
\displaystyle {\rm{subject}}\;{\rm{to}}\;\;\;\;\;\v{a} = \v{g}.
\end{array}
\end{equation}

Now consider a \textbf{competitive equilibrium},
$(\v{a},\v{g},\v{p})$, at which the vector $\v{a}$ solves the consumer's optimization
problem \eqref{eq:exa_C}, the vector $\v{g}$ solves the supplier's optimization
problem \eqref{eq:exa_S}, and the market clears, i.e., $\v{a}=\v{g}$.
Suppose that the utility functions are concave and continuously
differentiable, and that the cost functions $C$ and $H$ are convex and
continuously differentiable. We further assume that $H'(0)=0$,
 and that for $t=0,1$, $U'_t(0)>C'(0)$, $U'_t(B)<C'(B)$. Then, there exists a competitive
equilibrium, $(\v{a},\v{g},\v{p})$, which satisfies the following
conditions:
\begin{equation}\label{eq:exa_C1}
\left\{ \begin{array}{l}
\displaystyle U'_0({a_0}) = {p_0},\\[5pt]
\displaystyle U'_1({a_1}) = {p_1},
\end{array} \right. \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\left\{ \begin{array}{l}
\displaystyle C'({a_0}) -H'(a_1-a_0) = {p_0},\\[5pt]
\displaystyle C'({a_1}) +H'(a_1-a_0)= {p_1}.
\end{array} \right.
\end{equation}
We conclude that the competitive equilibrium solves the social
welfare maximization problem in \eqref{eq:exa_W}, because it satisfies the following
(sufficient) optimality conditions:
\begin{equation}\label{eq:exa_C2}
\begin{array}{l}
 \displaystyle U'_0({a_0})   = C'(a_0)-H'(a_1-a_0),\qquad  U'_1({a_1})  = C'(a_1)+ H'(a_1-a_0),\\[5pt]
 \displaystyle a_0=g_0,\;\;\;a_1=g_1.
 \end{array}
\end{equation}

However, we observe that the socially optimal price $p_0$ does not
equal the supplier's {instantaneous} marginal cost at stage $0$,
$C'(a_0)$. Hence, by setting the price equal to $C'(a_0)$, {as would
be done in a real-time balancing market,} we may not achieve social
optimality. More generally, marginal cost pricing need not be
socially optimal because it does not take into account the
externality conferred by the action $a_0$ on the ancillary cost at
stage $1$, $H(a_1-a_0)$. At a socially optimal competitive
equilibrium, the consumer should pay
$$
\big(C'(a_0)-H'(a_1-a_0)\big)a_0+\big(C'(a_1)+H'(a_1-a_0)\big)a_1,
$$
{i.e.,} the price on $a_0$ {should be} the sum of the supplier
marginal cost at stage $0$, $C'(a_0)$, and the marginal ancillary
cost associated with $a_0$, $-H'(a_1-a_0)$, {which is determined at
the next stage, after $a_1$ is realized}. } $\;\;\;\;\;\;\square$
\end{example}

Before {describing the precise} pricing mechanism {we propose}, we
introduce a differentiability assumption on the cost functions.

\begin{assumption}\label{A:price}
For any $s \in \mathcal {S}$, $C(\cdot,s)$ and $H_0(\cdot,s)$ are continuously
differentiable on ${[0},\infty)$.
 For any $(A',\overline s) \in \mathcal {A} \times \mathcal {S}^2$, $H(A,A',\overline s)$ and $H(A',A,\overline s)$
 are continuously differentiable in $A$ on $[0,\infty)$.\footnote{At the boundary of
the domain, $0$, we require continuity of the
right-derivatives of $C$, $H_0$, and $H$.}
 \end{assumption}

Inspired by Example \ref{ex:price}, we introduce prices
\begin{equation}\label{equa:pricep}
{p_t} =C'(A_t, s_t ),\qquad t=0,\ldots,T,
\end{equation}
and
\begin{equation}\label{equa:priceq}
q_t =  \dfrac{\partial H(A_{t-1}, A_t, \overline s_t )}{\partial
A_{t-1}},\quad w_t =  \dfrac{\partial H(A_{t-1}, A_t, \overline s_t
)}{\partial A_{t}},  \qquad t=1,\ldots,T.
\end{equation}
At stage $0$, we let $q_0=0$ and $w_0=H_0'(A_0,s_0)$.
 Under the proposed pricing mechanism, consumer $i$'s payoff at stage $t$ is
given by
\begin{equation}\label{equa:Spayoff1}
U_t(x_{i,t},s_t,a_{i,t})  - (p_t+w_t) a_{i,t} -   q_t a_{i,t-1} .
\end{equation}
Note that $p_t+w_t$ is the supplier marginal cost at stage $t$
(including the marginal ancillary cost). The proposed pricing
mechanism charges consumer $i$ an additional price $q_t$ {on her
previous demand}, equal to the marginal ancillary cost with respect
to $a_{i,t-1}$.

We now {define some of the notation that we will be using.}
 For $t=1,\ldots,T$, let
$y_{i,t}=(a_{i,t-1},x_{i,t})$ be the \textbf{augmented state} of
consumer $i$ at stage $t$. At $t=0$,
 let $y_{i,0}=x_{i,0}$.
{For} stage $t$, let $\mathcal {Y}_t$ be the set of all possible
augmented states. In particular, we have $\mathcal {Y}_0=\mathcal
{X}_0$, and $\mathcal {Y}_t=\mathcal {A} \times \mathcal {X}_t$, for
$t=1,\ldots,T$.

{Let $\Delta_n(D)$ be} the set of empirical probability
distributions over a given set {$D$} that can be generated by $n$
samples from $D$. (Note that empirical distributions are always
discrete, even if $D$ is a continuous set.) Let $f_{t} \in
\Delta_n(\mathcal {Y}_t)$ {be} the empirical distribution of the
augmented state of all consumers at stage $t$, and {let} $f_{-i,t}
\in \Delta_{n-1}(\mathcal {Y}_t)$ {be} the empirical distribution of
the augmented state of all consumers (excluding consumer $i$) at
stage $t$. {We} refer to $f_t$ as the \textbf{population state} at
stage $t$. Let $u_{t} \in \Delta_n(\mathcal{A})$ denote the
empirical distribution of all consumers' actions at stage $t$, and
{let} $u_{-i,t} \in \Delta_{n-1}(\mathcal {A})$ be the empirical
distribution of all consumers' (excluding consumer $i$) actions at
stage $t$.

For a given $n$, it can be seen from (\ref{equa:pricep}) and
(\ref{equa:priceq}) that the prices, and thus the stage payoff in
(\ref{equa:Spayoff1}), are determined by the current global state,
{$\overline{s}_t$,} consumer $i$'s current augmented state,
{$y_{i,t}$,} and current action, {$a_{i,t}$,} as well as the
empirical {distributions, $f_{-i,t}$ and $u_{-i,t}$} of other
consumers' current augmented state and action. Hence, for a certain
function $\pi(\cdot)$, we can write the stage payoff in
(\ref{equa:Spayoff1}) as
\begin{equation}\label{equa:Spayoff}
\pi(y_{i,t},\overline s_t,a_{i,t},f_{-i,t},u_{-i,t}) =
U_t(x_{i,t},s_t,a_{i,t})  - (p_t+w_t) a_{i,t} -   q_t a_{i,t-1} .
\end{equation}

\section{A Continuum Model and Dynamic Oblivious Strategies}\label{sec:DOE}

To study the aggregate behavior of a large number of consumers, we
consider a nonatomic game {involving} a continuum of infinitesimally
small consumers, indexed by $ i \in [0,1]$. We assume that (under state $s_0$) a fraction $\eta_{s_0}$
of the consumers has initial state $x$.
In a nonatomic model,
any single consumer's action has no influence on the aggregate
demand and the prices.
 We consider a class of {strategies (dynamic oblivious
strategies)} in which a consumer's action depends only on the
history of past exogenous states, $h_t=(s_0,\ldots,s_t)$, and her
{own} current state\footnote{Note that a dynamic oblivious strategy
depends only on the consumer's current state,
  instead of her augmented state. As we will see in Section \ref{sec:DOE2},
 in a continuum model, since any single consumer has no influence on the prices,
a best response or equilibrium strategy need not take into account
the action taken at the previous stage.}, i.e., of the form
$$
a_{i,t}=  \overline \nu_{t}( x_{i,t},  h_t).
$$
Suppose that consumer $i$ uses a dynamic oblivious strategy
$\overline \nu=(\overline \nu_0,\ldots,\overline \nu_T)$. Since
there is no idiosyncratic randomness, given a history $h_t$, the
state $x_{i,t}$ of consumer $i$ at stage $t$ depends only on her
initial state $x_{i,0}$. That is, there is a mapping $l_{\overline
\nu,h_t}: \mathcal {X}_0 \to \mathcal {X}_{t}$, such that
$x_{i,t}=l_{\overline \nu,h_t}(x_{i,0})$. Therefore, we can
{specify} the action taken by a dynamic oblivious strategy {in the
alternative form}
 \begin{equation}\label{equa:OE_stra}
a_{i,t}=  \nu_{t}( x_{i,0},  h_t)  \buildrel \Delta \over =
\overline \nu_{t}( l_{\overline \nu,h_t}(x_{i,0}), h_t) .
 \end{equation}
{We refer to} $\nu=(\nu_0,\ldots,\nu_T )$ {as a {\bf dynamic
oblivious strategy}, and let} $\mathfrak{V}$ be the set of all such
strategies.

An alternative {formulation involving strategies that depend on
consumer} expectations on future prices would lead to a Rational
Expectations Equilibrium (REE), an equilibrium concept based on the
rational expectations approach pioneered by \cite{M61}. In our
continuum model, since the only source of stochasticity is from the
exogenous state $s_t$, future prices {under any given strategy
profile,} are completely determined by the history $h_t$. Therefore,
it is {reasonable} to expect that {strategies of the form}
\eqref{equa:OE_stra} will lead to an equilibrium concept that
is identical in outcomes with a REE (cf.\ the discussion in Section
\ref{sec:DOE2}).

Before formally defining a Dynamic Oblivious Equilibrium (DOE), we
{first provide some of the intuition behind the definition.}
 In a continuum model, if all consumers use a common dynamic oblivious strategy $\nu$,
  the aggregate demand and the prices at stage $t$ depend only on the {history} of exogenous states,
$h_t=(s_0,\ldots,s_t)$.  A dynamic oblivious strategy $\nu$ is a DOE
(cf. the formal definition in Section \ref{sec:DOE2}) if it
maximizes every consumer's  {expected total payoff}, under the
sequence of prices that $\nu$ induces. In Section \ref{sec:DOE3}, we
associate a continuum model with a sequence of $n$-consumer models
($n=1,2,\ldots$),
 and
specify the relation between the continuum model and the corresponding $n$-consumer model.

\subsection{The sequence of prices induced by a dynamic oblivious
strategy}\label{sec:DOE1}

Let $h_t=(s_0,\ldots,s_t)$ denote a history up to stage $t$, and
let $\mathcal {H}_t = \mathcal {S}^{t+1}$ denote the set of all
possible such histories. Recall that in a continuum model,
given an initial global state $s_0$, the distribution of consumers'
initial states  is $\eta_{s_0} $. Therefore, under a history $h_t$,
if all consumers use the same dynamic oblivious strategy $\nu$, then
the average demand is
\begin{equation}\label{equa:OE_demand}
\widetilde A_{t \mid \nu, h_{t}}  =  \sum\limits_{x \in \mathcal
{X}_0 } \eta_{s_0} (x) \cdot \nu_{t}(x,h_{t}).
\end{equation}

We now introduce the cost functions in a continuum model. Let
$\widetilde C: \mathbb{R} \times \mathcal {S} \to [0,\infty)$ be a
primary cost function. Let $\widetilde H: \mathbb{R}^2 \times
\mathcal {S}^2 \to [0,\infty)$ be an ancillary cost function at
stage $t \ge 1$, and let $\widetilde H_0: \mathbb{R} \times \mathcal
{S} \to [0,\infty)$ be an ancillary cost function at the initial
stage~$0$.

Given the cost functions in a continuum model, we define the
sequence of prices induced by a dynamic oblivious strategy as
follows:
\begin{equation}\label{equa:OE_price0}
\widetilde p_{t|\nu,h_t} =\widetilde C'(\widetilde
A_{t|\nu,h_t},s_t),\qquad\widetilde q_{0|\nu,h_0}=0, \qquad
\widetilde w_{0|\nu,h_0}= \widetilde H'_0(\widetilde
A_{0|\nu,h_0},s_0),
\end{equation}
and for $t \ge 1$,
\begin{equation}\label{equa:OE_price}
\widetilde q_{t|\nu,h_t}=\dfrac{ \partial \widetilde H\left(
\widetilde A_{t-1|\nu,h_{t-1}}, \widetilde A_{t|\nu,h_{t}},
\overline s_t \right) } { \partial \widetilde
A_{t-1|\nu,h_{t-1}}},
\qquad
\widetilde w_{t|\nu,h_t}=\dfrac{ \partial
\widetilde H\left( \widetilde A_{t-1|\nu,h_{t-1}}, \widetilde
A_{t|\nu,h_{t}}, \overline s_t \right) } { \partial \widetilde
A_{t|\nu,h_{t}}}.
\end{equation}

\subsection{Equilibrium strategies} \label{sec:DOE2}

In this subsection we define the concept of a DOE. Suppose that
all consumers other than $i$ use a dynamic oblivious strategy $\nu$.
In a continuum model, consumer $i$'s action does not affect the
prices. If all consumers except $i$ use a dynamic oblivious strategy
$\nu$,
 consumer $i$'s \textbf{oblivious stage value} (the stage payoff in a continuum model) under a history $h_t$ and an action $a_{i,t}$, is
 \begin{equation}\label{equa:OE_Cpayoffa}
\widetilde \pi_{i,t}(y_{i,t}, h_t, a_{i,t} \mid
 {\nu} )   =
 U_t(x_{i,t},s_t,a_{i,t})  - (\widetilde p_{t|\nu,h_t}+ \widetilde w_{t|\nu,h_t}) a_{i,t} -   \widetilde
 q_{t|\nu,h_t}
 a_{i,t-1},
\end{equation}
where the prices, $\widetilde p_{t|\nu,h_t},$ $\widetilde
w_{t|\nu,h_t}$,  and $\widetilde q_{t|\nu,h_t}$, are defined in
\eqref{equa:OE_price0} and \eqref{equa:OE_price}. Since a single
consumer's action cannot influence $\widetilde q_{t}$, the last term
in \eqref{equa:OE_Cpayoffa} is not affected by the action $a_{i,t}$,
and the decision $a_{i,t}$ at stage $t$ need not take $a_{i,t-1}$
into account, but should take $\widetilde {q}_{t+1}$ into account.

 Consumer $i$'s oblivious stage
value  under a dynamic oblivious strategy $\hat \nu$,
is\footnote{{Recall that} the initial state (the type) of consumer
$i$, $x_{i,0}$, is included in its state $x_{i,t}$, for any $t$.}
 \begin{equation}\label{equa:OE_Cpayoff}
\widetilde \pi_{i,t}(y_{i,t}, h_t  \mid \hat \nu, {\nu} ) \buildrel
\Delta \over = \widetilde \pi_{i,t}(y_{i,t}, h_t, \hat
\nu_t(x_{i,0}, h_t ) \mid
 {\nu} ).
\end{equation}
In particular, we use $\widetilde \pi_{i,t}(y_{i,t},  h_t \mid \nu,
{\nu} ) $ to denote the oblivious stage value of consumer $i$ at
stage $t$, if all consumers use the strategy $\nu$. Given {an
initial global state $s_0$  and an initial state of consumer $i$,
$x_{i,0}$,} her \textbf{oblivious value function} ({total} future
expected payoff function in a continuum model) is
 \begin{equation}\label{equa:OE_Cvalue}
 \widetilde V_{i,{0}} ( x_{i,{0}}, {s_0}  \mid \hat
\nu ,{\nu} ) =
 \mathbb{E}\left\{ {\sum\limits_{\tau={0}}^T { \widetilde
\pi_{i,\tau}(y_{i,\tau},  h_\tau \mid \hat \nu, {\nu} )} } \right\},
 \end{equation}
where the expectation is over the future global states, $\{s_\tau
\}_{\tau=1}^T$.

\begin{definition}\label{Def:OE}
A strategy $\nu$ is a \textbf{Dynamic Oblivious Equilibrium} (DOE)
if
\[
\sup _{\widehat \nu  \in \mathfrak{V} } \;\widetilde V_{i,0}(
x_{i,0}, s_0  \mid \hat \nu, {\nu}) = \widetilde V_{i,0}( x_{i,0},
s_0  \mid  \nu, {\nu}),\qquad\forall x_{i,0} \in \mathcal {X}_{0},
\quad\forall s_0 \in \mathcal {S}.
\]
\end{definition}

A DOE is guaranteed to exist, under suitable assumptions, and this
is known to be the case for our model (under our assumptions),
and even for a more general
model that includes idiosyncratic randomness \citep{BB92}.
 The DOE, as defined above, is
essentially the same concept as the ``dynamic competitive
equilibrium'' studied in \cite{BC10}, which is defined as the
nonatomic equivalent of an MPE, in a continuum model. At a DOE, the
beliefs of all consumers on future prices are consistent with
the equilibrium outcomes. Therefore, a DOE is identical in outcomes
with a Rational Expectations Equilibrium (REE).

In future electricity markets, consumers may form rational
expectations of future prices through an adaptive learning process,
or they may receive price estimates from utilities and/or the
independent system operator through advanced metering
infrastructures. (In Appendix \ref{sec:im}, we {provide} some
discussion {of a possible} implementation of the proposed real-time
pricing mechanism.) {If so, a REE (equivalently, a DOE) will be a
plausible outcome of such a market.} Furthermore, we will show
(Theorem \ref{thm:SO_social}) that under the proposed pricing
mechanism, and under certain convexity assumptions, a DOE is
socially optimal for the continuum model.

For a  dynamic market model with aggregate profit shocks,
\cite{WBV10} introduce a concept of  ``extended oblivious
equilibrium'' at which every firm's strategy depends on its current
state and on the recent history (as opposed to the full history) of
the aggregate shock. The extended oblivious equilibrium is
computationally tractable; however, an equilibrium strategy
 may not be an approximate best response for every firm, even if the
number of firms is large (cf. the error bounds derived in Section
8.3 of \cite{WBV10}).

 Note that the definition of a DOE strategy
requires optimality (attaining the supremum in Definition
\ref{Def:AMP}) only along the equilibrium path \citep{BC10}. Thus, a
DOE is similar in spirit to the ``self-confirming equilibria'' in
\cite{FL93} and the ``subjective equilibria''  in \cite{KL95}, in
which each agent forms correct beliefs about her opponents only
along the equilibrium path.

\subsection{The $n$-consumer model associated with
a continuum model}\label{sec:DOE3}

 We want
the cost functions in a continuum model to approximate the cost
functions in an $n$-consumer model. Since the continuum of consumers
is described by distributions over $[0,1]$, the demand given in
\eqref{equa:OE_demand} can be regarded as the average demand per
consumer. {To capture this correspondence,} we assume the following
relation between the cost functions in a continuum model and their
counterparts in a corresponding $n$-consumer model.

\begin{assumption}\label{A:scale}
For any $n \in \mathbb{N}$, any $s \in \mathcal {S}$, and any
$\overline s$ in $ \mathcal {S}^2$, we have
$$
C^n(A,s) =n \widetilde C \left( \frac{A}{n}, s \right), \;\;\;
 H_0^n(A,s) =n \widetilde H_0 \left( \frac{A}{n}, s \right), \;\;\;
H^n(A,A',\overline s) =n \widetilde H \left( \frac{A}{n},
\frac{A'}{n}, \overline s\right),
$$
where the superscript $n$ is used to indicate that these are the
cost functions associated with an $n$-consumer model.
\end{assumption}

Assumption \ref{A:scale} implies that
$$
(C^n)'(A,s) =\widetilde C'(A/n,s),\qquad (H_0^n)'(A,s) =\widetilde
H_0'(A/n,s),\quad s \in \mathcal {S},
$$
and
$$
\dfrac{\partial H^n(A,A',\overline s)}{\partial A} =\dfrac{\partial
\widetilde H(A/n,A'/n,\overline s)}{\partial
(A/n)},\qquad\dfrac{\partial H^n(A,A',\overline s)}{\partial A'}
=\dfrac{\partial \widetilde H(A/n,A'/n,\overline s)}{\partial
(A'/n)},\;\;\;\forall \overline s \in \mathcal {S}^2,
$$
 i.e., the prices in the continuum model at
the average demand equal the prices \jnt{in} the corresponding
 $n$-consumer model.

\section{Approximation in Large Games} \label{sec:AMP}
In this section, we  consider a sequence of dynamic games, and show that as the
number of consumers increases to infinity,
 a DOE strategy for the corresponding continuum game is asymptotically optimal for every
consumer (i.e., an approximate best response), if the other
consumers follow that same strategy. In the rest of the paper, we
{often} use a superscript $n$ to indicate quantities associated with
an $n$-consumer model.

Suppose that all consumers except $i$ use a dynamic oblivious
strategy $\nu$.  Given a history $h_t$ and an empirical distribution
$f^n_{-i,t}$,
   we use  $v (h_t, f^n_{-i,t}, \nu)$ to denote the empirical
 distribution, {$u^n_{-i,t}$,} of the actions taken by consumers excluding $i$.
In an $n$-consumer model,  suppose that consumer $i$ uses a
{\bf history-dependent strategy} $\kappa^n=\{ \kappa^n_{t} \}_{t=0}^T$ of
the form
\begin{equation}\label{eq:his}
a_{i,t}=\kappa^n_{t}(y_{i,t},h_t,f^n_{-i,t}),
\end{equation}
 while  the other
consumers use a dynamic oblivious strategy $\nu$. {Let}
$\mathfrak{K}_n$ denote the set of all possible history-dependent
strategies $\kappa^n$ for the $n$-consumer model. Note that since
all other consumers use an oblivious strategy $\nu$, $f^n_{-i,t}$ is
completely determined by $\nu$, $f^n_{-i,0}$, and $h_t$.

 The stage payoff received by consumer $i$
at time $t$ is
 \begin{equation}\label{equa:AMP_payoff}
\pi^n_{i,t}( y_{i,t}, h_t, f^n_{-i,t} \mid \kappa^n, {\nu} )= \pi^n
\left(  y_{i,t}, \overline s_t, a_{i,t}, f^n_{-i,t}, v( h_t,
f^n_{-i,t},  {\nu} ) \right),
\end{equation}
where $a_{i,t}=\kappa^n_{t}(y_{i,t},h_t,f^n_{-i,t})$, and the stage
payoff function on the right-hand side is given in
(\ref{equa:Spayoff}).
 Given an initial global state, $s_0$, and consumer $i$'s initial
 state, $x_{i,0}$,
 consumer $i$'s expected payoff under the strategy
$\kappa^n$ is
 \begin{equation}\label{equa:AMP_value}
 V_{i,0} ^n \left(x_{i,0}, s_0 \mid \kappa^n, {\nu}
\right) =  \mathbb{E}\left\{ {\sum\limits_{t=0}^T { \pi_{i,t}^n(  y_{i,t},
h_t, f^n_{-i,t} \mid \kappa^n, {\nu} )} } \right\},
\end{equation}
where the  expectation is over the
 initial distribution  $f^n_{-i,0}$ and over the
future global states, $\{s_t \}_{t=1}^T$. In particular, we use
$V_{i,0} ^n \left(x_{i,0}, s_0 \mid \nu, {\nu} \right)$ to denote
the expected payoff obtained by consumer $i$ if all consumers use
the strategy $\nu$.

\begin{definition}\label{Def:AMP}
A dynamic oblivious strategy $\nu$ has the \textbf{asymptotic Markov
equilibrium (AME)} property \citep{AJ10}, if for any initial global
state $s_0 \in \mathcal {S}$, any initial consumer state $x_{i,0}
\in \mathcal {X}_0 $, and any sequence of history-dependent
strategies $\{\kappa^n\}$, we have
\[
\limsup_{n \to \infty } \left( V_{i,0} ^n \left(x_{i,0}, s_0 \mid
\kappa^n, {\nu}   \right) - V_{i,0} ^n \left( x_{i,0},s_0
 \mid \nu, {\nu} \right) \right) \le 0.
\]
\end{definition}

We will show that every DOE has the
AME property, under the following assumption, which strengthens Assumption \ref{A:price}.

\begin{assumption}\label{A:continuous}
We assume that:
\begin{itemize}
\item [{\rm \ref{A:continuous}.1.}] The {following} four families of functions, {of $A$}, $\{\widetilde
C'({A},s): \; s \in \mathcal{S}\}$, $\{\widetilde H'_0({A}, s): \; s
\in \mathcal{S}\}$, $ \{{\partial \widetilde H(A, A', \overline s)}/{\partial
A}: \; (A',\overline s) \in \mathcal {A} \times \mathcal {S}^2\}$,
and $\{{\partial \widetilde H(A', A, \overline s )}/{\partial A}:
\;(A',\overline s) \in \mathcal {A} \times \mathcal {S}^2\}$, are uniformly
equicontinuous on $[0,\infty)$.\footnote{A sufficient condition for
this assumption to hold is to require a universal bound on the
derivatives of the functions in each family.}

\item [{\rm \ref{A:continuous}.2.}] The marginal costs are bounded from above, i.e.,
 $$
 |\widetilde C'(A,s)| \le P, \qquad  |\widetilde H_0'(A,s)| \le P, \quad \forall (A,s) \in \mathcal {A} \times \mathcal {S},
 $$
 and
 $$
\left|\frac{\partial \widetilde H(A, A', \overline s )}{\partial
A}\right| \le P,\qquad
\left|\frac{\partial \widetilde H(A', A,
\overline s )}{\partial A}\right| \le P,\quad \forall (A',\overline
s) \in \mathcal {A} \times \mathcal {S}^2,
 $$
where $P$ is a positive constant.

\old{\item The transition probability function, $p(x,x',a,s)$, is
continuous in $a$, for any $(x,x',s) \in \mathcal {X}^2 \times
\mathcal {S}. $}

\item [{\rm \ref{A:continuous}.3.}] The utility functions, $\{ U_t(x,s,a) \}_{t=0}^T$, are continuous in $a$ and bounded above, i.e.,
 $$
 U_t(x,s,a) \le Q, \qquad t=0,\ldots,T, \qquad \forall (x,s,a) \in \mathcal {X}_t \times \mathcal {S} \times \mathcal {A},
 $$
where $Q$ is a positive constant.
\end{itemize}
\end{assumption}

Combining with Assumption \ref{A:scale}, Assumption \ref{A:continuous}.1 implies
that  for any \(\varepsilon
>0\), there exists a $\delta>0$ such that for any positive integer
$n$, if $|A-\overline A| \le n \delta$, then
\begin{equation}\label{equa:Acon}
\left|  (C^n)'(A,s) -   (C^n)'(\overline A,s) \right| \le
\varepsilon,\qquad\left|  (H_0^n)'(A,s) -   (H_0^n)'(\overline A,s)
\right| \le \varepsilon,\quad\forall s \in \mathcal {S},
\end{equation}
and for any $(A',\overline s) \in \mathcal {A} \times \mathcal
{S}^2$,
\begin{equation}\label{equa:Acon1}
\left|  \frac{\partial H^n(A, A', \overline s )}{\partial A} -
\frac{\partial H^n(\overline A, A', \overline s )}{\partial
\overline A}\right| \le \varepsilon,\qquad \left|  \frac{\partial
H^n(A', A, \overline s)}{\partial A} - \frac{\partial H^n(
A',\overline A, \overline s )}{\partial \overline A}\right| \le
\varepsilon.
\end{equation}

Note that the boundness of the cost function derivatives implies the
Lipschitz continuity of the cost functions. Combining with
Assumption \ref{A:scale}, for any pair of real numbers $(A,\overline
A)$, and any positive integer $n$, we have
\begin{equation}\label{equa:Alip}
\left|  C^n(A,s) -   C^n(\overline A,s) \right| \le P|A-\overline
A|,\qquad\left| H_0^n(A,s) -   H_0^n(\overline A,s) \right| \le
P|A-\overline A|,\quad\forall s \in \mathcal {S},
\end{equation}
and for any $(A',\overline s) \in \mathcal {A} \times \mathcal
{S}^2$,
\begin{equation}\label{equa:Alip1}
\left|  H^n(A, A', \overline s ) -   H^n(\overline A, A', \overline
s )\right| \le P|A-\overline A|,\qquad  \left| H^n(A', A,  \overline
s ) - H^n(A', \overline A,  \overline s ) \right| \le P|A-\overline
A|.
\end{equation}

\old{Suppose that all consumers other than $i$ use a DOE strategy
$\nu$. We argue (Lemma \ref{L:DP_mu}) that as the number of
consumers increases to infinity, the maximum expected payoff
consumer $i$ can obtain is asymptotically no larger than the optimal
oblivious value, {and that the optimal oblivious value} can be
approximately achieved if she uses the DOE strategy (Lemma
\ref{L:DP_nu}).}

 We argue in the following theorem that a DOE
strategy approximately maximizes a consumer's expected payoff (among
all possible history-dependent strategies) in a dynamic game with a
large but finite number of consumers, if the other consumers also
use that strategy.

\begin{theorem}\label{thm:APE}
Suppose that Assumptions \ref{A:scale}-\ref{A:continuous} hold.
Every DOE has the AME property.
\end{theorem}

Theorem \ref{thm:APE} is proved in Appendix  \ref{sec:A}. Various
approximation properties of nonatomic equilibrium concepts in a
continuum game have been investigated in previous works.
 Sufficient conditions for a
stationary equilibrium (an equilibrium concept for a continuum game
without aggregate uncertainty) to have the AME property are derived
in \cite{AJ10}. For a continuum game with both idiosyncratic and
aggregate uncertainties,  \cite{BC10} shows that as the number
agents increases to infinity, the actions taken in an MPE can be
well approximated by some DOE strategy of the nonatomic limit game.
Note, however, that in a {general} $n$-consumer game, even if all
consumers take an action that is close to the action taken by a DOE
strategy of the nonatomic limit game, the population states and the
prices in the $n$-consumer game can still be very different from
their counterparts in the nonatomic limit game. Therefore, without
further assumptions on the consumers' state transition kernel (e.g., continuous dependence of consumer states on their previous actions),
the
approximation property of a DOE on the action space
 does not necessarily imply the AME property of the DOE.

\section{Asymptotic Social Optimality} \label{sec:social}
In Section \ref{sec:social1}, we define the social welfare
associated with an $n$-consumer model and with a continuum model. In
Section \ref{sec:social2},  we show that for a continuum model, the
social welfare is maximized (over all symmetric dynamic oblivious
strategy profiles) at a DOE, and that for a sequence of $n$-consumer
models, if all consumers use the DOE strategy of the corresponding
continuum model,
  then the
social welfare is asymptotically maximized, as the number of consumers increases to infinity.

\subsection{Social welfare}\label{sec:social1}

In an $n$-consumer model, let {$\v{x}_t=(x_{1,t},\ldots,x_{n,t})$
and $\v{a}_t=(a_{1,t},\ldots,a_{n,t})$ be the vectors of consumer
states and actions, respectively,} at stage $t$. Under the current
{global state $\overline s_t$,} the
 social welfare realized at stage $t$ is
\begin{equation}\label{equa:SO_wel}
 W^n_t(\v{x}_{t},\overline s_t,\v{a}_t) = -C^n(A_t,s_t) -
 H^n(A_{t-1},A_t,\overline s_t) +\sum\limits_{i=1}^n
U_t(x_{i,t},s_t,a_{i,t}),\qquad t=1,\ldots,T,
\end{equation}
and at stage $0$, the social welfare is
\begin{equation}\label{equa:SO_wel0}
 W^n_t(\v{x}_0, s_0,\v{a}_0) = -C^n(A_0,s_0) -
H_0^n(A_0, s_0) +\sum\limits_{i=1}^n U_0(x_{i,0},s_0,a_{i,0}).
\end{equation}

{Because of the symmetry of the problem, the social welfare at stage
$t$ depends on $\v{x}_{t}$ and $\v{a}_t$ only through the empirical distribution of state-action pairs.
In particular, under a symmetric history-dependent
strategy profile $\v{\kappa}^n=(\kappa^n,\ldots,\kappa^n)$ (cf. the
definition of a history-dependent strategy in Eq. \eqref{eq:his}),
we can write the social welfare at time $t$ (with a slight abuse of
notation) as} $W^n_t(f^n_{t},h_t \mid \v{\kappa}^n)$. Given an
initial global state $s_0$ and an initial population state $f^n_0$,
the expected social welfare achieved under a symmetric
history-dependent strategy profile $\v{\kappa}^n$ is given by
\begin{equation}\label{equa:SO_welfarenu}
\mathcal {W}^n_{0}(f^n_0,s_0 \mid \v{\kappa}^n) = W^n_0(f^n_0,s_0
\mid \v{\kappa}^n) +
 \mathbb{E}\left\{ \sum\limits_{t = 1}^T { W^n_t(f^n_t,h_t\mid
\v{\kappa}^n }) \right\},
\end{equation}
where  the expectation is over the future global states
$\{s_t\}_{t=1}^T$. In particular, we use $\mathcal
{W}^n_{0}(f^n_0,s_0 \mid \v{\nu}^n)$ to denote the expected social
welfare achieved by the ``symmetric dynamic oblivious strategy profile'',
$\v{\nu}^n=(\nu,\ldots,\nu)$.

In a continuum model, suppose that all consumers use a {common}
dynamic oblivious strategy $\nu$.
Given an initial global state $s_0$, the expected social welfare is
\begin{equation}\label{equa:SO_CWel}
\widetilde {\mathcal {W}}_0 (s_0 \mid {\nu})=\widetilde W_0( s_0
\mid \nu )+ \mathbb{E}\left\{ \sum\limits_{t=1}^T \widetilde W_t(
h_t \mid \nu) \right\},
\end{equation}
where the expectation is over the future global states,
$\{s_t\}_{t=1}^T$. Here, $\widetilde W_t( h_t \mid \nu)$ is the
stage social welfare under history $h_t$:
\begin{equation}\label{equa:SO_CWelt}
\displaystyle
\begin{array}{l}
 \displaystyle \widetilde W_t( h_t \mid \nu ) =
  -\widetilde C( \widetilde A_{t|\nu,h_t},s_t) -
  \widetilde H(\widetilde A_{t-1|\nu,h_{t-1}},\widetilde A_{t|\nu,h_t},\overline s_t)
  \\[3pt]
\displaystyle \;\;\;\;\;\;\;   \;\;\;\;\;\;\; \;\;\;\;
\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;
\;\;\;\;\; \;\;\;\;\;\; + \sum\limits_{x \in  \mathcal {X}_0 }
\eta_{s_0} (x)
  U_t\left(l_{\nu,h_t}(x),s_t,\nu_t(x, h_t)
  \right),\;\;\;\;\;\;\; \;\;\;t=1,\ldots,T,
\end{array}
\end{equation}
where $l_{\nu,h_t}$ maps a consumer's initial state into her state
at stage $t$, under the history $h_t$ and the dynamic oblivious
strategy $\nu$. The social welfare at stage $0$ is given by
\begin{equation}\label{equa:SO_CWelt0}
 \widetilde W_0(s_0 \mid \nu ) =
  -\widetilde C( \widetilde A_{0|\nu,h_0},s_0) -
  \widetilde H_0(\widetilde A_{0|\nu,h_0}, s_0)
  + \sum\limits_{x \in  {\mathcal {X}}_{0} } \eta_{s_0} (x)
  U_0\left(x,s_0,\nu_0(x,s_0)
  \right).
\end{equation}

 \subsection{Asymptotic social optimality of a DOE}\label{sec:social2}

\old{
\jnt{[IS THIS USEFUL?] Let us start with some perspective on the connection between the continuum and the $n$-consumer model. Let us fix a dynamic oblivious strategy $\nu$ and suppose that it is used by all consumers in the $n$-consumer model, resulting in the social welfare $\mathcal
{W}^n_{0}(f^n_0,s_0 \mid \v{\nu}^n)$. It is not hard to show that as $n\to\infty$, the expected value (over the initial population state  $f^n_0$, which is drawn from the distribution $\eta_{s_0}$) of the social welfare per user, $\mathcal
{W}^n_{0}(f^n_0,s_0 \mid \v{\nu}^n)/n$, converges to the social welfare $\widetilde {\mathcal {W}}_0 (s_0 \mid {\nu})$ under $\nu$ in the continuum model.[IS THIS CORRECT?] The proof (omitted) relies on the laws of large numbers, together with the correspondence
between the two models introduced in Section \ref{sec:DOE3}. Part (b) of Theorem \ref{thm:SO_social} below will be in the same spirit, except that it will make a comparison between with a more general class of strategy profiles (i.e., history-dependent) for the $n$-consumer model and will focus on the issue of optimality.}
}

We now define some notation that will be useful in this subsection.
Since there is no idiosyncratic randomness, given a history $h_t$,
the state of consumer $i$ at stage $t$ depends only on her initial
state $x_{i,0}$, and her actions taken at $\tau=0,\ldots,t-1$.
 At stage $t \ge 1$, the history $h_t$ and the transition function $z_{i,t+1}=r(x_{i,t},a_{i,t},s_{t+1})$ define
a mapping $k_{h_t} : \mathcal {X}_0 \times \mathcal {A}^t  \to
\mathcal {Z}$:
\begin{equation}\label{equa:SO_k}
z_{i,t}=k_{h_t}(x_{i,0},a_{i,0},\ldots,a_{i,t-1}),\;\;\;t=1,\ldots,T.
\end{equation}
Given an initial state $x_{i,0}$,  consumer $i$'s total utility
under a history $h_t$ can be written as a function of her actions
taken at stages $\tau=0,\ldots,t$:
\begin{equation}\label{equa:SO_U}
\overline U_{h_t}(x_{i,0},a_{i,0},\ldots,a_{i,t}) =
U_t(x_{i,0},s_0,a_{i,0})+ \sum\limits_{\tau=1}^t
U_t(x_{i,0},k_{h_\tau}(x_{i,0},a_{i,0},\ldots,a_{i,\tau-1}),s_\tau,a_{i,\tau}).
\end{equation}

Before proving the main result of this section, we introduce a
series of assumptions on the convexity and differentiability of the
cost and the utility functions.

\begin{assumption}\label{A:convex}
We assume the following.
\begin{itemize}

\item [{\rm \ref{A:convex}.1.}] For any $s \in \mathcal {S}$, $\widetilde C(\cdot,s)$ is convex;
for any $\overline s \in \mathcal {S}^2$,  $\widetilde
H(A,A',\overline s)$ is convex in $(A,A')$.

\item [{\rm\ref{A:convex}.2.}] For any $h_T \in \mathcal {H}_T$ and any $x_{i,0} \in \mathcal
{X}_0$, the function defined in (\ref{equa:SO_U}) is concave with
respect to the vector $(a_{i,0},\ldots,a_{i,T})$.

\item [{\rm \ref{A:convex}.3.}] For any $t \ge 1$, any $h_t \in \mathcal {H}_t$, and any $x_{i,0} \in \mathcal
{X}_0$, the function $k_{h_t}$ defined in \eqref{equa:SO_k} is monotonic in
$a_{i,\tau}$, for $\tau=0,\ldots,t-1$; further, its left and right
derivatives with respect to  $a_{i,\tau}$ exist, for
$\tau=0,\ldots,t-1$.

\item [{\rm \ref{A:convex}.4.}] For $t \ge 1$, and for any $(x,s,a) \in \mathcal {X}_0 \times \mathcal {S} \times \mathcal {A} $,
 the left and right derivatives of the utility function $U_t(x,z,s,a)$ in $z$ exist.
\end{itemize}
\end{assumption}

Assumption \ref{A:convex}.1 is standard. If the utility function is
concave in $a$, Assumption \ref{A:convex}.2 requires that the
transition function $k_{h_t}$ preserves concavity (a linear function would be an example).
 Note that Assumptions
\ref{A:convex}.1 and \ref{A:convex}.2  guarantee that in both models
(a dynamic game with a finite number of consumers, and the
corresponding continuum game), the expected social welfare (consumer
$i$'s expected payoff) is concave in the vector of actions taken by all
consumers (respectively, by consumer $i$).
 Assumptions \ref{A:convex}.3 and
\ref{A:convex}.4 ensure the existence of left and right derivatives
of the expected social welfare given in (\ref{equa:SO_CWel}), with
respect to the actions taken by consumers. An example where
Assumptions \ref{A:convex}.2-\ref{A:convex}.4  hold is given next.

\begin{example}{\rm
Consider appliances such as Plug-in Hybrid Electric
Vehicles (PHEVs), dish washers, or clothes washers. For such
appliances, a customer usually only cares  whether a task is completed
before a certain time.

 Given an initial state (type) of consumer
$i$, $x_{i,0}$, let $D(x_{i,0})$ and $T(x_{i,0})$ indicate her total
desired demand and the stage {by which} the task has to be
completed, respectively. Under a given history $h_t$, the total
utility {accumulated} by consumer $i$ until time $t$ is assumed to
be of the form
$$
\overline U_{h_t}(x_{i,0},a_{i,0},\ldots,a_{i,t}) = Z\left(x_{i,0},
\min\left\{ D(x_{i,0}), \sum\limits_{\tau=0}^{\min\{T(x_{i,0}),t\}}
a_{i,\tau} \right\} \right),
$$
for some function $Z$. If for every $x_{i,0} \in \mathcal {X}_0$,
$Z(x_{i,0},\,\cdot\,)$ is nondecreasing and concave, then Assumption
\ref{A:convex}.2 holds. At stage $t=0$, we have
$$
U_{0}(x_{i,0},s_0,a_{i,0})=Z\left( x_{i,0}, \min\left\{ D(x_{i,0}),
a_{i,0} \right\} \right).
$$
For $t=1,\ldots, T(x_{i,0})$, we {let} $z_{i,t}=
\sum\nolimits_{\tau=0}^{t-1} a_{i,\tau}$, and
$$
U_t(x_{i,0},z_{i,t},s_t,a_{i,t})=Z\left( x_{i,0}, \min\left\{
D(x_{i,0}), a_{i,t}+ z_{i,t} \right\} \right)-Z\left( x_{i,0},
\min\left\{ D(x_{i,0}), z_{i,t} \right\} \right) .
$$
For $t \ge T(x_{i,0})+1$, we let $z_{i,t}= D(x_{i,0})$, and let $
U_t(x_{i,t},s_t,a_{i,t}) $ be identically zero. Suppose that for
every $x_{i,0} \in \mathcal {X}_0$, the right and left derivatives
of $Z(x_{i,0},\,\cdot\,)$ exist. Then, Assumptions \ref{A:convex}.3 and
\ref{A:convex}.4 hold. }$\;\;\;\square$
\end{example}

\begin{theorem}\label{thm:SO_social}
Suppose that Assumptions \ref{A:scale}-\ref{A:convex} hold. Let
$\nu$ be a DOE of the continuum game. Then, the following hold.
\begin{itemize}
\item[(a)] In the continuum game, the social welfare is maximized
{(over all symmetric dynamic oblivious strategy profiles)} at the
DOE, i.e., \footnote{{Note that we are only comparing the social
welfare under different {\bf symmetric} dynamic oblivious strategy
profiles, where all consumers are using the same dynamic oblivious
strategy ($\nu$ or $\vartheta$). This is no loss of generality
because} under Assumption \ref{A:convex},
  the social welfare in a continuum game is {a concave function of the collection of consumer actions taken under the} different histories.
  Hence, {it can be shown that}
 the optimal social welfare can be achieved by a symmetric dynamic oblivious strategy
profile.}
$$
  \widetilde {\mathcal W}_0(s_0 \mid {\nu})  = \sup\nolimits_{ \vartheta \in \mathfrak{V}}
   \widetilde {\mathcal W}_0(s_0 \mid \vartheta)  ,
 \;\;\; \forall s_0 \in \mathcal {S},
$$
where $\mathfrak{V}$ is the set of all dynamic oblivious strategies.

\item[(b)] For a sequence of $n$-consumer games, the symmetric DOE strategy profile, $\v{\nu}^n=(\nu,\ldots,\nu)$, approximately maximizes the
 expected social welfare, as the number of consumers increases to infinity. That is,
 for any
initial global state $s_0$, and any sequence of symmetric
history-dependent strategy profiles $\{\v{\kappa}^n\}$, we
have\footnote{Under Assumption \ref{A:convex},  the social welfare
in an $n$-consumer game is {a concave function of the collection of
consumer  actions taken under the} different histories. Therefore,
$\sup_{{\kappa}^n \in \mathfrak{K}_n} \mathcal {W}^n_{0}(f^n_0,s_0
\mid \v{\kappa}^n) $ is also the maximum social welfare that can be
achieved by a (possibly non-symmetric) history-dependent strategy
profile. }
\[
 \limsup_{n \to \infty } \mathbb{E}\left\{
\dfrac{   \mathcal {W}^n_{0}(f^n_0,s_0 \mid \v{\kappa}^n)-\mathcal
{W}^n_{0}(f^n_0,s_0 \mid \v{\nu}^n)}{n}   \right\} \le 0,
\]
where the expectation is over the initial population state, $f^n_0$.
\end{itemize}

\end{theorem}

The proof of Theorem \ref{thm:SO_social} is given in Appendix
\ref{sec:C}.

\section{Conclusion and Future Directions}\label{sec:con}

In an electric power system, load swings may result in significant
ancillary cost to suppliers. {Motivated} by the observation that
  marginal cost pricing  may not achieve social optimality in
electricity markets, we proposed a new dynamic pricing mechanism
that takes into account the externality conferred by a consumer's
action on future ancillary cost. Besides proposing a suitable
game-theoretic model that incorporates the cost of load fluctuations
and
 a particular pricing mechanism for electricity markets, a main contribution of this paper
was to show that the proposed pricing mechanism achieves social
optimality in a dynamic nonatomic game, and approximate social
optimality for the case of finitely many consumers, under certain
convexity assumptions.

To compare the proposed pricing mechanism with marginal cost
pricing, we presented a numerical example in which the demand
increases sharply at the last stage. In this example,  the proposed
pricing mechanism creates a stronger incentive for consumers to
shift their peak load than marginal cost pricing, through an
additional negative price charged on {off-peak consumer demand.} As
a result, compared with marginal cost pricing, the proposed pricing
mechanism achieves a higher social welfare, and at the same time,
reduces the peak load, and therefore has the potential to reduce the
need for long-term investments in peaking plants.

We believe that the constructed dynamic game-theoretic model, the
proposed pricing mechanism, and more importantly, the insights
provided by this work, can be applied to a more general class of
markets with friction.  As an extension and future work, one can
potentially develop and use variations of our framework to a market
of a perishable product/service where demand fluctuations incur
significant cost to suppliers. {Examples include} data centers
implementing cloud services that suffer from the switching costs to
toggle a server into and out of a power-saving mode \citep{LW11},
and large organizations such as hospitals that use on-call staff to
meet unexpected demand.

\bibliographystyle{nonumber}


\section{Introduction}
The nonlinear dynamics of electrons in carbon nanotubes (CNs) under the action of an external electric field has been the subject of intense research \cite{1, 4}. Tans {\it et al.} \cite{1} and Bezryadin {\it et al.} \cite{2} have measured the  current-voltage characteristics for single-wall CNs at low temperatures, i.e when $k_BT < \epsilon_C$, $\Delta \epsilon$. Where $k_B$ is Boltzmann constant, $T$ is the temperature, $\epsilon_C$ is the charging energy. The energy level spacing $\Delta \epsilon$ is given by $\Delta \epsilon = \pi \hbar v_F/L$ where $v_F$ is the Fermi velocity and L is the carbon nanotube length \cite{4}. At this low temperatures the current is produced by the electrons tunneling through the CNs in the presence of the Coulomb blockade induced by long-range Coulomb interactions \cite{4}. Nonlinear coherent transport through doped nanotube junctions was considered in Ref. \cite{3} and it also showed the possibility of negative differential conductivity (NDC) for tunelling electrons. Using semiclassical Boltzmann’s equation, Maksimenko {\it et al.}l \cite{4} found that the dc current- voltage characteristics of CNs biased by a constant electric field along the axis of undoped CNTs at room temperatures ( $k_BT > \epsilon_C$, $\Delta\epsilon$ ) showed NDC up to frequencies on the order of the Bloch frequency. Here the current density is produced by conduction electrons with energies below the energy of the interband transitions and move in the crystalline field like free quasiparticles, with a modified dispersion law allowing the use of quasiclassical approach to describe the electron motion. Bloch oscillations have been observed in semiconductor superlattices \cite{5, 6}. These Bloch oscillations are caused by Bragg reflection of electrons at the edges of the Brillouin regions and the occurrence of Bloch oscillations are concluded from the observation of NDC.

Like Maksimenko {\it et al.} \cite{4}, quantum phenomena like interband transitions, quantum mechanical corrections to intraband motion as well as Coulomb interactions will be excluded from this report. Our report will be quasiclassical with an extension to the high-frequency electric field in CNs by following the approach of Ktitorov {\it et al.}  \cite{7}.

\section{Theoretical model}
Following \cite{4}, the high-frequency differential conductivity is derived starting with the Boltzmann kinetic equation with \cite{5}. For simplicity, we shall ignore the difference between the energy - momentum relaxation rates and assume a common relaxation time $\tau$ for both processes. The equations of the symmetric $f_s$ and antisymmetric $f_a$ parts of the distribution functions are
\begin{equation}
	\frac{\partial f_s}{\partial t} + eE(t)\frac{\partial f_a}{\partial k_x} = - \frac{f_s - f_0(p)}{\tau} \label{eq:one}
\end{equation}
\begin{equation}
	\frac{\partial f_a}{\partial t} + eE(t)\frac{\partial f_s}{\partial k_x} = - \frac{f_a}{\tau} \label{eq:two}
\end{equation}
Writing the distribution functions in Fourier series as
\begin{equation}
	f_a^1 = \Delta p_{\varphi} \sum_{s=1}^{n}\delta(p_{\varphi} - s\Delta p_{\varphi})\sum_{r\neq 0} f_{rs}e^{iarp_z}\Phi_a^1 \label{eq:three}
\end{equation}
\begin{equation}
	f_s^1 = \Delta p_{\varphi} \sum_{s=1}^{n}\delta(p_{\varphi} - s\Delta p_{\varphi})\sum_{r\neq 0} f_{rs}e^{iarp_z}\Phi_s^1 \label{eq:four}
\end{equation}
\begin{equation}
	f_0(p) = \Delta p_{\varphi} \sum_{s=1}^{n}\delta(p_{\varphi} - s\Delta p_{\varphi})\sum_{r\neq 0} f_{rs}e^{iarp_z} \label{eq:five}
\end{equation}
where $e$ is the electron charge, $f_0(p)$ is the equilibrium distribution function, $\delta(x)$ is the Dirac delta function, $r$ is summation over the stark component, $ f_{rs}$ is the coefficient of the Fourier series and $\Phi_s^1$ and $\Phi_a^1$ are the factors by which the Fourier transform of the symmetric $f_s^1$ and antisymmetric $f_a^1$ parts of the distribution functions are different from the equilibrium distribution function $f_0(p, t)$, and $\tau_e = \tau_p = \tau$ is the relaxation time. The electric field $E$  is applied along CNs axis.  The equilibrium distribution function can be expanded in the analogous series with the coefficients $f_{rs}$ expressed as
 \begin{equation}
 	  f_{rs} = \frac{a}{2\pi}\int_0^{\frac{2\pi}{a}} \frac{e^{iarp_z}}{1 + exp(\epsilon_s(p_z))/k_BT)} \label{eq:six}
\end{equation}

The investigation is done within the semiclassical approximation in which conduction electrons with energy below the energy of the interband transitions move in the crystalline lattice like free quasi-particles with dispersion law extracted from quantum theory \cite{4}. Taking into account the hexagonal crystalline structure of a rolled graphene in a form of CNT and using the tight binding approximation, the energy dispersion for zigzag and armchair CNTs for which the  Fermi energy $\epsilon_F=0$, are  expressed as in Eqns \eqref{eq:seven} and \eqref{eq:eight}, respectively \cite{4, 8, 9}
 \begin{eqnarray}
	\epsilon_s(s\Delta p_{\varphi}, p_z) &\equiv& \epsilon_s(p_z)\nonumber\\
	 &=& \pm\gamma_0\Big[ 1 + 4 cos (ap_z) cos\Big(\frac{a}{\sqrt{3}}s\Delta p_\varphi\Big) \ldots\nonumber \\
	 && + 4cos^2\Big( \frac{a}{\sqrt{3}}s\Delta p_{\varphi}\Big)\Big]^{1/2}\label{eq:seven}
\end{eqnarray}
\begin{eqnarray}
	\epsilon_s(s\Delta p_{\varphi}, p_z) &\equiv& \epsilon_s(p_z)\nonumber\\
	 &=& \pm\gamma_0\Big[ 1 + 4 cos (as\Delta p_{\varphi}) cos\Big(\frac{a}{\sqrt{3}} p_z\Big) \ldots\nonumber\\
	 && + 4 cos^2\Big(\frac{a}{\sqrt{3}} p_z\Big)\Big]^{1/2}\label{eq:eight}
\end{eqnarray}
where $\gamma_0 \sim 3.0$ eV is the overlapping integral, $p_z$ is the axial component of quasi-momentum, $\Delta p_{\varphi}$ is transverse quasi-momentum  level spacing and s is an integer. The expression for a in Eqns \eqref{eq:seven} and \eqref{eq:eight} is given as  $a=3b⁄2\hbar$, $b=0.142$ nm is the C-C bond length. The – and + signs correspond to the valence and conduction bands respectively. Due to the transverse quantization of the quasi-momentum, its transverse component can take n discrete values, $p_{\varphi} = s\Delta p_{\varphi} = \pi\sqrt{3}s⁄an$   ($s=1, \ldots, n$). Unlike transverse quasi-momentum $p_{\varphi}$, the axial quasi momentum   is assumed to vary continuously within the range $0 \leq p_z \leq 2\pi⁄a$, which corresponds to the model of infinitely long  of CNT ($L = \infty$).

The quasiclassical velocity $v_z(p_z, s∆p_{\phi})$ of an electron moving along the CNT axis is given by the expression
\begin{equation}
	v_z (p_z, s∆p_z ) = \gamma_0 \sum_{r \neq 0} \frac{\partial (\epsilon_{rs} e^{iarp_z} )}{\partial p_z } = \gamma_0 \sum_{r  \neq 0} iar \epsilon_{rs} e^{iarp_z}     \label{eq:nine}
\end{equation}

Substituting Eqns \eqref{eq:three}, \eqref{eq:four} and \eqref{eq:five} into Eqns \eqref{eq:one} and \eqref{eq:two} and solving with the perturbations $f_s = f_s^0 + f_s^1 e^{-i\omega t}$, $f_a = f_a^0 + f_a^1 e^{-i\omega t}$ and $E = E_0 + E_1 e^{-i \omega t}$, we obtain the surface current density defined by
\[
	j_z = \frac{2e}{(2\pi \hbar)^2}\int\int f(p,t)  v_z (p) d^2 p
\]
or
\begin{equation}
	j_z = \frac{2e}{(2\pi \hbar)^2}\sum_{s=1}^{n}\int_0^{\frac{2\pi}{a}} f\big(p_z, s\Delta p_{\varphi}, \Phi(t) \big)  dp_z \label{eq:ten}
\end{equation}
as
\begin{eqnarray}
	\sigma_z (\omega) &=& \sigma_0 \frac{4e^2\gamma_0\sqrt{3}}{n\hbar^2} \sum_{r=1}r\left[ \frac{1 - i\omega\tau^2 - \omega_B^2 \tau^2}{(\omega_B^2 - \omega^2) \tau^2 + 1 - 2i\omega\tau} \right] \nonumber \\
		&& \times \sum_{s=1}^n f_{rs} \epsilon_{rs} \label{eq:eleven}
\end{eqnarray}
\begin{equation}
               \sigma_0 = \frac{\sigma_0 (0)}{(\omega_B^2\tau^2 + 1) }  \label{eq:twelve}
\end{equation}

Here, $j_0 = \frac{4 e^2 \gamma_0 \sqrt{3}}{n\hbar^2}$. $\omega_B = eaE/\sqrt{3}$ for armchair and $\omega_B = eaE_0$ for zigzag CNs.
\begin{figure}[ht!]
\includegraphics[scale=.8]{LAD1.eps}
\caption{Real (red) and imaginary (blue) parts of conductivity for armchair CNs for $\omega/\omega_B = 10$.}\label{fig:one}
\end{figure}

\begin{figure}[ht!]
\includegraphics[scale=.8]{LAD2.eps}
\caption{Real (red) and imaginary (blue) parts of conductivity for zigzag CNs for $\omega/\omega_B = 10$.}\label{fig:two}
\end{figure}

\section{Results, Discussion and Conclusion}
Using quasiclassical approach, we consider an influence of Bragg reflections on dynamics of an electron transport in a CNs simultaneously exposed to both constant (dc) and ac electric fields. An analytical expression for the current density is obtained. The nonlinearity is analyzed basically on the dependence of the current density on the frequency.

In Figs.\ref{fig:one} and \ref{fig:two}, we show the plot of the real and imaginary parts of the normalized current density ($j_z/j_0$) as a function of dimensionless frequency ($\omega/\omega_B$) for a-SWCNT and z-SWCNT, respectively. In all two systems, we observed that $Re(\sigma_z)$ is negative for $\omega < 1\tau  \sqrt{(\omega_B \tau )^4 - 1) /( (\omega_B\tau)^2 - 1)}$ which is indicative for gain at these frequencies.

In conclusion,    we have large amplitude current dynamics due to Bragg reflections of carbon nanotubes effect in undoped CNs. We have obtained an expression for the current density-electric field characteristics for CNs in presence of dc-ac fields. We noted that the high density of states of conduction electrons in metallic carbon nanotubes and the specific dispersion law inherent in hexagonal crystalline structure result in a uniquely high gain at these frequencies than the corresponding values for semiconducting ones.