text:
# TR-RB simulation

```python
# ~~~
# This file is part of the paper:
#
#           "An adaptive projected Newton non-conforming dual approach
#         for trust-region reduced basis approximation of PDE-constrained
#                           parameter optimization"
#
#   https://github.com/TiKeil/Proj-Newton-NCD-corrected-TR-RB-for-pde-opt
#
# Copyright 2019-2020 all developers. All rights reserved.
# License: Licensed as BSD 2-Clause License (http://opensource.org/licenses/BSD-2-Clause)
# Authors:
#   Luca Mechelli (2019 - 2020)
#   Tim Keil      (2019 - 2020)
# ~~~
```

# Preparations

## details

In this notebook we discretize the following problem:
Find $u_{\mu}$ solving the following parametrized constrained equation

\begin{align*}
	-  \nabla \cdot \left( \kappa_{\mu}  \nabla u_{\mu} \right) &= f_{\mu} &&\text{in } \Omega, \\
	q ( \kappa_{\mu}  \nabla u_{\mu} \cdot n) &= (u_{\text{out}} - u_{\mu}) &&\text{on } \partial \Omega.
\end{align*}

For the definition of $\kappa_{\mu}$ and $f_{\mu}$ in the computational domain $\Omega$, we use the `BitmapFunction` from PyMOR. This function uses the Python Imaging Library (PIL) in order to convert a .png grayscale picture to a data function on our domain.
Walls, windows and doors are stored in seperated .png to enable a affine decomposition of each part of the picture. Furthermore, we are considering a heater at each window. The numbering of the components for $\kappa_{\mu}$ can be seen in the following picture

Here, numbers with | are walls, numbers with _ are doors and numbers with dots are windows

The heaters are numbered in the following way:

For our pde constrained model, we need to define a cost functional. In a general quadratic model, we have

\begin{align}
\mathcal{J}(u, \mu) := \Theta(\mu) + j_\mu(u) + k_\mu(u, u),
\end{align}

For this code, we restrict ourselves to the following definition
\begin{align}
\mathcal{J}(v, \mu) = \frac{\sigma_d}{2} \int_{D}^{} (v - u^{\text{d}})^2 + \frac{1}{2} \sum^{M}_{i=1} \sigma_i \mu_i^2,
\end{align}
which means

\begin{align}
\Theta(\mu) &= \frac{1}{2} \sum^{M}_{i=1} \sigma_i \mu_i^2 + \frac{\sigma_d}{2} \int_{D}^{} u^{\text{d}} u^{\text{d}}\\
j_{\mu}(u) & = -\sigma_d \int_{D}^{} u^{\text{d}}u \\
k_{\mu}(u,u) &= \frac{\sigma_d}{2} \int_{D}^{} u^2
\end{align}

The following code implements this particular case.

## prepare cells

```python
import numpy as np

from matplotlib import pyplot as plt

from pymor.basic import *
set_log_levels({'pymor': 'WARN'})
```

```python
from pymor.core.logger import set_log_levels, getLogger
set_log_levels({'pymor': 'ERROR',
                'distributed_adaptive_discretizations': 'DEBUG',
                'notebook': 'INFO'})
logger = getLogger('notebook.notebook')
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (12.0, 8.0)
mpl.rcParams['font.size'] = 12
mpl.rcParams['savefig.dpi'] = 300

data_path = '../../../EXC_data'
# domain of interest
bounding_box = [[0,0],[2,1]]
domain_of_interest = BitmapFunction('{}/Domain_of_interest.png'.format(data_path), range=[1,0], bounding_box=bounding_box)
```

## problem definition and tolerances

```python
from pdeopt.problems import EXC_problem, set_input_dict
from pdeopt.discretizer import discretize_quadratic_pdeopt_stationary_cg

parametric_quantities = {'walls': [1,4,9], 'windows': [], 'doors': [6,7], 'heaters': [1,3,5,6,7,8,9]}
inactive_quantities = {'removed_walls': [], 'open_windows': [], 'open_doors': [1,2,3,4,5,10], 'active_heaters': []}
summed_quantities = {'walls': [[1,2,3,7,8],[4,5,6]], 'windows': [], 'doors': [], 'heaters': [[1,2],[3,4],[9,10,11,12]]}

coefficient_expressions = None

parameters_in_q = True
input_dict = set_input_dict(parametric_quantities, inactive_quantities, coefficient_expressions, summed_quantities, parameters_in_q,
                            ac=0.5, owc=[0.025,0.1], iwc= [0.025,0.1], idc=[0.05,0.2], wc=[0.0005], ht=[0,100],
                                    owc_c=0.001,  iwc_c= 0.025,     idc_c=0.01,  wc_c=0.05,  ht_c=80)

parameter_scaling = False
u_out = 5

problem, parameter_scales = EXC_problem(input_dict, summed_quantities, outside_temperature=u_out, #, q_inverse=0.0001
                                        data_path = data_path,parameters_in_q=parameters_in_q,
                                        parameter_scaling=parameter_scaling,
                                        coefficient_expressions=coefficient_expressions)

u_d = 18

mu_d = problem.parameter_space.sample_randomly(1, seed=10)[0]
mu_d = None

sigma_d = 100
weights = {'walls': 0.1, 'doors': 1, 'heaters': [0.002,0.002,0.0005,0.0005,0.0005,0.0005,0.004], 'windows': 1, 'state': sigma_d}

diameter = np.sqrt(2)/200.
opt_fom, data, mu_bar = discretize_quadratic_pdeopt_stationary_cg(problem, diameter, weights, parameter_scales,
                                                          domain_of_interest, desired_temperature=u_d,
                                                          mu_for_u_d=mu_d, mu_for_tikhonov=mu_d,
                                                          parameters_in_q=parameters_in_q, product='fixed_energy')
```

    I am using the NCD corrected functional!!
    my product is fixed_energy
    mu_bar is: {doors: [0.1, 0.1], heaters: [10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0], walls: [0.049999999999999996, 0.049999999999999996, 0.049999999999999996]}

```python
print('Information on the grid:')
print(data['grid'])

seed = 7                   # random seed for the starting value
radius = 0.1               # TR radius
FOC_tolerance = 1e-7       # tau_FOC
sub_tolerance = 1e-8       # tau_sub
safety_tol = 1e-16         # Safeguard, to avoid running the optimizer for really small difference in digits
max_it = 60                # Maximum number of iteration for the TR algorithm
max_it_sub = 400           # Maximum number of iteration for the TR optimization subproblem
max_it_SH = 800            # Maximum number of iteration for the TR Steihaug method
max_it_arm = 50            # Maximum number of iteration for the Armijo rule
init_step_armijo = 0.5     # Initial step for the Armijo rule
armijo_alpha = 1e-4        # kappa_arm
beta = 0.95                # beta_2
epsilon_i = 1e-8           # Treshold for the epsilon active set (Kelley '99)
iterative_solver = 'CG'    # chosen iterative solver

# parameter control
control_mu = False         # enable post processing for the optimal parameter estimation

# optional enrichment
tau_g = FOC_tolerance/sub_tolerance
tau_grad = 0.01
beta_3 = 1./2
minimum_radius = 1e-6
maximum_radius = 1e16

# some implementational variables
Qian_Grepl_subproblem = True
reductor_type = 'simple_coercive'
```

    Information on the grid:
    Rect-Grid on domain [0,2] x [0,1]
    x0-intervals: 400, x1-intervals: 200
    faces: 80000, edges: 160600, vertices: 80601

```python
# starting with
parameter_space = problem.parameter_space
mu = parameter_space.sample_randomly(1, seed=seed)[0]
```

### What methods do you want to test ?

```python
optimization_methods = [
    # FOM Method
    'SH',   # Steihaug Version 4
    # Newton
        # NCD-corrected
         'Method_4', # TR-RB (a)
         'Method_4_optional',
         'Method_5', # TR-RB (c)
         'Method_5_optional',
        # Aggregated
#          'Method_6', # TR-RB (b)
#          'Method_7', # TR-RB (d)
    # BFGS
        # NCD-corrected from KMSOV'20
         'Method_8', # TR-RB
    # Qian Grepl method from QGVW'17
#          'Method_9'
]

#optimization_methods = ['All']
```

## Visualizations

Visualize diffusion and rhs

```python
from pymor.discretizers.builtin.cg import InterpolationOperator

vis_mu = mu.copy()
diff = InterpolationOperator(data['grid'], problem.diffusion).as_vector(vis_mu)
rhs = InterpolationOperator(data['grid'], problem.rhs).as_vector(vis_mu)
doI = InterpolationOperator(data['grid'], domain_of_interest).as_vector(vis_mu)

opt_fom.visualize(diff)
```

    Output()

```python
opt_fom.visualize(rhs)
```

    Output()

```python
opt_fom.visualize(doI)
```

    Output()

We can easily solve the primal and dual equations by calling

```python
u = opt_fom.solve(mu)
p = opt_fom.solve_dual(mu)

opt_fom.visualize(u)
```

    Output()

```python
opt_fom.visualize(p)
```

    Output()

# Full Order Optimization

```python
parameters = opt_fom.parameters
if mu_d is not None:
    mu_opt = mu_d
else:
    #use result from previous FOM run
    mu_opt = parameters.parse(np.array([5.000000000000e-02,  5.000000000000e-02,  2.238254711947e+01,
  2.339650455721e+01,  4.870348426512e+01,  4.937422778852e+01,
  5.236272251326e+01,  5.411556305764e+01,  2.352380079299e+01,
  2.500000000000e-02,  2.500000000000e-02,  2.500000000000e-02]))

mu_opt_as_array = mu_opt.to_numpy()
J_opt = opt_fom.output_functional_hat(mu_opt)
print('Optimal parameter: ', mu_opt_as_array)
print('Optimal J: ', J_opt)

print()
print('Starting parameter: ', mu.to_numpy())
J_start = opt_fom.output_functional_hat(mu)
print('Starting J: ', J_start)
```

    Optimal parameter:  [5.00000000e-02 5.00000000e-02 2.23825471e+01 2.33965046e+01
     4.87034843e+01 4.93742278e+01 5.23627225e+01 5.41155631e+01
     2.35238008e+01 2.50000000e-02 2.50000000e-02 2.50000000e-02]
    Optimal J:  5.813965062384796

    Starting parameter:  [6.14462434e-02 1.66987819e-01 4.38409231e+01 7.23465178e+01
     9.77989512e+01 5.38495870e+01 5.01120464e+01 7.20511334e+00
     2.68438980e+01 6.24911876e-02 7.59422497e-02 8.52804277e-02]
    Starting J:  142.0522187579113

```python
print(mu_opt.parameters)
```

    {doors: 2, heaters: 7, walls: 3}

# TR Steihaug

```python
from pdeopt.TR import TR_Steihaug
from pdeopt.tools import compute_errors

TR_parameters = {'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'max_iterations_subproblem': max_it_sub,
                 'max_iterations': max_it_SH,
                 'starting_parameter': mu,
                 'epsilon_i': epsilon_i,
                 'max_iterations_armijo': max_it_arm, 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'full_order_model': True,
                 'iterative_solver': iterative_solver}

if 'SH'  in optimization_methods or 'All' in optimization_methods:
    mus_SH, times_SH, Js_SH, FOC_SH = TR_Steihaug(opt_fom, problem.parameter_space, TR_parameters)
    times_full_SH, J_error_SH, mu_error_SH, FOC = compute_errors(opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array,
                                                       mus_SH, Js_SH, times_SH, 0, FOC_SH)
```

    Computing at first the Cauchy point
    The model function admits a point that has all components on the boundary
     Returning the direction to reach it to continue
    Using CG-Steihaug for the linear system
    Enlarging the radius to 0.2
    k: 1 - Cost Functional: 136.10745705465433 - mu: {doors: [0.050408879609046685, 0.05], heaters: [43.64092314408935, 72.14651778309413, 97.59895119966026, 53.64958704104336, 49.91204636599379, 7.005113335976154, 26.64389801018712], walls: [0.026339250055162355, 0.02681974525667649, 0.09947419142240559]}
    First order critical condition: 9.225556531973776
    ***********************************************

    Computing at first the Cauchy point
    The model function admits a point that has all components on the boundary
     Returning the direction to reach it to continue
    Using CG-Steihaug for the linear system
    Enlarging the radius to 0.4
    k: 2 - Cost Functional: 127.6363496661561 - mu: {doors: [0.05002948748842964, 0.05], heaters: [43.24092314408935, 71.74651778309412, 97.19895119966026, 53.24958704104336, 49.51204636599379, 6.605113335976154, 26.24389801018712], walls: [0.025096583736709355, 0.025131235982460073, 0.09996207985430194]}
    First order critical condition: 8.91185638984663
    ***********************************************

    Computing at first the Cauchy point
    The model function admits a point that has all components on the boundary
     Returning the direction to reach it to continue
    Using CG-Steihaug for the linear system
    Enlarging the radius to 0.8
    k: 3 - Cost Functional: 111.75954772963587 - mu: {doors: [0.05000430848349404, 0.05], heaters: [42.44092314408935, 70.94651778309412, 96.39895119966026, 52.449587041043365, 48.71204636599379, 5.805113335976154, 25.44389801018712], walls: [0.025014112067780817, 0.025019175185625006, 0.09999445940191816]}
    First order critical condition: 8.294575372212766
    ***********************************************

    Computing at first the Cauchy point
    The model function admits a point that has all components on the boundary
     Returning the direction to reach it to continue
    Using CG-Steihaug for the linear system
    Enlarging the radius to 1.6
    k: 4 - Cost Functional: 83.42930965969208 - mu: {doors: [0.05000129116970799, 0.05], heaters: [40.84092314408935, 69.34651778309413, 94.79895119966027, 50.849587041043364, 47.11204636599379, 4.205113335976154, 23.843898010187118], walls: [0.025004229115525408, 0.025005746434646488, 0.09999833958922733]}
    First order critical condition: 7.06071045029595
    ***********************************************

    Computing at first the Cauchy point
    The model function admits a point that has all components on the boundary
     Returning the direction to reach it to continue
    Using CG-Steihaug for the linear system
    Enlarging the radius to 3.2
    k: 5 - Cost Functional: 40.408091355104716 - mu: {doors: [0.05000081769564197, 0.05], heaters: [37.64092314408935, 66.14651778309413, 91.59895119966026, 47.64958704104336, 43.91204636599379, 1.005113335976155, 20.64389801018712], walls: [0.025002678291872187, 0.02500363920756368, 0.09999894846460206]}
    First order critical condition: 4.593203440104053
    ***********************************************

    Computing at first the Cauchy point
    The model function admits a point that has all components on the boundary
     Returning the direction to reach it to continue
    Using CG-Steihaug for the linear system
    Enlarging the radius to 6.4
    k: 6 - Cost Functional: 8.573480522572481 - mu: {doors: [0.05000084468492149, 0.05], heaters: [31.24092314408935, 59.74651778309413, 85.19895119966026, 41.24958704104336, 37.51204636599379, 1.0382886195179597, 14.243898010187118], walls: [0.025002766692940093, 0.025003759325105127, 0.09999891375708825]}
    First order critical condition: 0.16101098027495242
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    Enlarging the radius to 12.8
    k: 7 - Cost Functional: 7.767605728084845 - mu: {doors: [0.050000000015161195, 0.05], heaters: [28.78495104113043, 52.332985476945105, 82.79787077166395, 41.479984415858084, 38.188392682666574, 4.0792449153801975, 17.731812186430062], walls: [0.09999865387994732, 0.09999865389776465, 0.02500134615021475]}
    First order critical condition: 0.07502637663184465
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    Enlarging the radius to 25.6
    k: 8 - Cost Functional: 6.856003459625526 - mu: {doors: [0.05, 0.05], heaters: [26.282668313591913, 38.23719501855718, 78.60826177332352, 42.89055559868527, 40.53819522556589, 11.375785188357831, 24.83980383552026], walls: [0.1, 0.1, 0.025]}
    First order critical condition: 0.21595751667675145
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    Enlarging the radius to 51.2
    k: 9 - Cost Functional: 6.014954779340087 - mu: {doors: [0.05, 0.05], heaters: [24.593423776138692, 26.85324200047438, 64.95608551320663, 48.590666946540324, 49.41902659966662, 36.91985098581522, 22.260232963953282], walls: [0.1, 0.025000104019906724, 0.07495190836907309]}
    First order critical condition: 0.14864788242147872
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    Enlarging the radius to 102.4
    k: 10 - Cost Functional: 5.840655580482007 - mu: {doors: [0.05, 0.08015553801805088], heaters: [22.910407342050615, 23.972111975199287, 50.37990087579896, 49.33138089316656, 51.9088354972311, 52.3806571278503, 23.69248071061213], walls: [0.024999999999999994, 0.025, 0.025]}
    First order critical condition: 0.13068462640457956
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 11 - Cost Functional: 5.815475944735226 - mu: {doors: [0.05, 0.050000000000000086], heaters: [22.741191128848282, 23.789848795831986, 49.97608566307175, 49.12460663973753, 51.7456631831286, 52.4938397152199, 23.535119816806784], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 0.0025068190511950924
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 12 - Cost Functional: 5.813979436352611 - mu: {doors: [0.05, 0.05000000000009504], heaters: [22.386681214013663, 23.400825234531318, 48.71494746110397, 49.374627145720844, 52.36069493003882, 54.10642138997954, 23.532171910286344], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 0.003096012247285
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 13 - Cost Functional: 5.8139651283870535 - mu: {doors: [0.05, 0.05000000000000013], heaters: [22.383372767536045, 23.39736453302319, 48.71309606036338, 49.37281438032227, 52.35876230126091, 54.10445929899869, 23.525171614053722], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 1.6844412405710693e-05
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 14 - Cost Functional: 5.813965062967327 - mu: {doors: [0.05, 0.05], heaters: [22.382575167277608, 23.39653383868942, 48.7035647966349, 49.374233354013796, 52.36271259803366, 54.1155055919613, 23.52384851623729], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 1.9687035768601324e-05
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 15 - Cost Functional: 5.8139650623877515 - mu: {doors: [0.05, 0.05], heaters: [22.382554093267792, 23.396511795093463, 48.70355295789207, 49.37422180007067, 52.36270028384165, 54.11549311155291, 23.523804058656587], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 1.0887179933211341e-07
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 16 - Cost Functional: 5.8139650623859325 - mu: {doors: [0.05, 0.05], heaters: [22.382547285736017, 23.39650471909192, 48.703484805260835, 49.374227834170625, 52.362722463198445, 54.11556269864411, 23.523801099446874], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 1.2535189332815576e-07
    ***********************************************

    Computing at first the Cauchy point
    Using CG-Steihaug for the linear system
    k: 17 - Cost Functional: 5.813965062384796 - mu: {doors: [0.05, 0.05], heaters: [22.382547151562253, 23.39650457900615, 48.70348473013142, 49.37422776067046, 52.36272238482928, 54.11556261913792, 23.523800816221975], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 6.768044573038316e-10
    ***********************************************

    Stopping criteria fulfilled: FOC condition: 6.768044573038316e-10

```python
if 'SH' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_SH,J_error_SH, label='SH J')
    plt.legend()
```

```python
# compute the error in the target
from pymor.discretizers.builtin.cg import (L2ProductP1, L2ProductQ1, InterpolationOperator)
from pymor.discretizers.builtin.grids.referenceelements import square
from pymor.discretizers.builtin.grids.boundaryinfos import EmptyBoundaryInfo
if data['grid'].reference_element is square:
    L2_OP = L2ProductQ1
else:
    L2_OP = L2ProductP1
if mu_d is None:
    if 'SH' in optimization_methods or 'All' in optimization_methods:
        empty_bi = EmptyBoundaryInfo(data['grid'])
        u_d = InterpolationOperator(data['grid'], ConstantFunction(u_d,2)).as_vector()
        diff= u.to_numpy()-u_d.to_numpy()
        diff_= opt_fom.solution_space.from_numpy(diff)
        Restricted_L2_OP = L2_OP(data['grid'], empty_bi, dirichlet_clear_rows=False, coefficient_function=domain_of_interest)
        print("{}".format(Restricted_L2_OP.apply2(diff_,diff_)[0][0]))
        print("{}".format(Restricted_L2_OP.apply2(diff_,diff_)[0][0]/Restricted_L2_OP.apply2(u_d,u_d)))
```

    2.5736284076967575
    [[0.06806596]]

# TR-RB approaches

```python
import time
from pdeopt.model import build_initial_basis
from pdeopt.reductor import QuadraticPdeoptStationaryCoerciveReductor
from pdeopt.TR import TR_algorithm, TR_algorithm_with_optional_enrichment

from pymor.parameters.functionals import MinThetaParameterFunctional
ce = MinThetaParameterFunctional(opt_fom.primal_model.operator.coefficients, mu_bar)

set_defaults({'pymor.operators.constructions.induced_norm.raise_negative': False})
set_defaults({'pymor.operators.constructions.induced_norm.tol': 1e-20})
```

## Newton Method

### NCD-corrected

#### Lagrange Basis (Method 4)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=True)
opt_fom = opt_fom.with_(adjoint_approach=True)

if 'Method_4' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                prepare_for_hessian=True,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'NewtonMethod',  'iterative_solver': iterative_solver,
                 'JFE_start': J_start} # This is not cheating because the quantity to compute this are accessible when preparing pdeopt_reductor
                                       # so the computational time to compute this is already

    extension_params = {'Enlarge_radius': True, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_4, times_4, Js_4, FOC_4, data_4 = TR_algorithm(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_4_actual, J_error_4_actual, mu_error_4_actual, FOC_4_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_4, Js_4, times_4, tictoc, FOC_4)
```

    Starting with two bases. Primal and dual have length 1 and 1
    building simple coercive primal reductor...
    building simple coercive dual reductor...
    starting parameter {doors: [0.06144624340609358, 0.1669878188360172], heaters: [43.84092314408935, 72.34651778309413, 97.79895119966027, 53.849587041043364, 50.11204636599379, 7.205113335976154, 26.843898010187118], walls: [0.062491187561917, 0.07594224970907054, 0.08528042770782818]}
    Starting value of the cost: 142.05221871073422
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.09707274] 8.386141954198166
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 2 and 2
    k: 0 - j 4 - Cost Functional: 113.12019599231121 - mu: {doors: [0.05, 0.07165965759763548], heaters: [41.3617810419687, 69.38608805942935, 95.24934022880507, 53.108086570853885, 51.78677857986843, 11.154448723073822, 24.689931699023266], walls: [0.025, 0.025, 0.1]}
    enlarging the TR radius to 0.2
    computing the fom gradient since the model was enriched
    First order critical condition: 8.359383634977865
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.19485449] 5.436167975475768
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 3 and 3
    k: 1 - j 4 - Cost Functional: 52.42160383261739 - mu: {doors: [0.05, 0.05], heaters: [34.88613223893655, 62.961312737330125, 90.1633066155099, 50.72682423638229, 55.974891139117915, 19.91458614202501, 17.569650053273897], walls: [0.025, 0.025, 0.1]}
    enlarging the TR radius to 0.4
    computing the fom gradient since the model was enriched
    First order critical condition: 5.435168320471611
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.39903053] 2.701587613005128
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 4 and 4
    enlarging the TR radius to 0.8
    k: 2 - j 3 - Cost Functional: 17.97344017207331 - mu: {doors: [0.05, 0.05], heaters: [29.933400540210265, 50.73487721387974, 76.66933501212476, 47.37442946711639, 54.10549819580599, 31.060338232313825, 16.777019160578313], walls: [0.026197497656193284, 0.03546662240938889, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 2.7063796812009207
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.78366649] 0.9922678308402169
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 5 and 5
    enlarging the TR radius to 1.6
    k: 3 - j 6 - Cost Functional: 7.550758090439643 - mu: {doors: [0.05, 0.05], heaters: [25.681703313322114, 35.40552760950831, 61.00629445313868, 48.24214443460455, 52.9034095564115, 43.70280954182427, 20.197821013914773], walls: [0.05959950378313627, 0.1, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.9929574402461987
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    CG truncated at iteration: 0 with residual: 0.54148, because p_k is not a descent direction
    Choosing the gradient as direction
    Using CG for the linear system
    CG truncated at iteration: 1 with residual: 0.05270, because p_k is not a descent direction
    Using CG for the linear system
    CG truncated at iteration: 1 with residual: 0.22345, because p_k is not a descent direction
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.010454493133116548, J_diff = 0.0005138220029902442, FOC = 3.552713678800501e-15
    relative differences mu 0.010454493133116548 and J 0.0005138220029902442
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 6 and 6
    k: 4 - j 7 - Cost Functional: 5.814067152854811 - mu: {doors: [0.05, 0.05], heaters: [22.417563101082013, 23.553481111884583, 48.785797686897695, 49.181429460254826, 51.97399026564293, 54.29607302982171, 23.52971265562549], walls: [0.025, 0.025, 0.025]}
    enlarging the TR radius to 3.2
    computing the fom gradient since the model was enriched
    First order critical condition: 0.0036581914188549424
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.004752155974444116, J_diff = 1.615701115372057e-05, FOC = 0.0
    relative differences mu 0.004752155974444116 and J 1.615701115372057e-05
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 7 and 7
    k: 5 - j 2 - Cost Functional: 5.81396524171808 - mu: {doors: [0.05, 0.05], heaters: [22.38297178919087, 23.393225184830648, 48.70727582517131, 49.388585310239705, 52.370127537440446, 54.09655070347377, 23.523731521196268], walls: [0.025, 0.025, 0.025]}
    enlarging the TR radius to 6.4
    computing the fom gradient since the model was enriched
    First order critical condition: 5.877394777836345e-05
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.00023137428403711196, J_diff = 3.026338365582005e-08, FOC = 3.889218348125276e-11
    relative differences mu 0.00023137428403711196 and J 3.026338365582005e-08
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 8 and 8
    k: 6 - j 2 - Cost Functional: 5.8139650628891815 - mu: {doors: [0.05, 0.05], heaters: [22.382540249497715, 23.396509836205237, 48.70345782678151, 49.37420763676373, 52.362755502301376, 54.11554819476222, 23.52381071513161], walls: [0.025, 0.025, 0.025]}
    enlarging the TR radius to 12.8
    computing the fom gradient since the model was enriched
    First order critical condition: 5.459640765978054e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.0, J_diff = 0.0, FOC = 9.93085844921655e-08
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 9 and 9
    k: 7 - j 6 - Cost Functional: 5.8139650628827315 - mu: {doors: [0.05, 0.05], heaters: [22.38254561408377, 23.396505689849405, 48.70347846121417, 49.37422338503675, 52.362729715619565, 54.11555978719734, 23.523802914232395], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 1.0059844153500897e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.0, J_diff = 0.0, FOC = 9.516489610666632e-08
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 10 and 10
    k: 8 - j 3 - Cost Functional: 5.813965062886204 - mu: {doors: [0.05, 0.05], heaters: [22.382545610297953, 23.396505685212503, 48.7034784593056, 49.374223383003525, 52.36272971268731, 54.115559784827695, 23.523802904423704], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 9.64746170192378e-08
    *******************************

    Stopping criteria fulfilled: FOC condition: 9.64746170192378e-08
    Computing only the initial FOC

```python
if 'Method_4' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4_actual,mu_error_4_actual)
```

#### Lagrange Basis with optional enrichment (Method 4.a)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=True)
opt_fom = opt_fom.with_(adjoint_approach=True)

if 'Method_4_optional' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                prepare_for_hessian=True,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'NewtonMethod',  'iterative_solver': iterative_solver,
                 'tau_g': tau_g, 'tau_grad': tau_grad, 'beta_3': beta_3,
                 'minimum_radius': minimum_radius, 'maximum_radius': maximum_radius,
                 'JFE_start': J_start}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_4bis, times_4bis, Js_4bis, FOC_4bis, data_4bis = TR_algorithm_with_optional_enrichment(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_4bis_actual, J_error_4bis_actual, mu_error_4bis_actual, FOC_4bis_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_4bis, Js_4bis, times_4bis, tictoc, FOC_4bis)
```

    Starting with two bases. Primal and dual have length 1 and 1
    building simple coercive primal reductor...
    building simple coercive dual reductor...
    starting parameter {doors: [0.06144624340609358, 0.1669878188360172], heaters: [43.84092314408935, 72.34651778309413, 97.79895119966027, 53.849587041043364, 50.11204636599379, 7.205113335976154, 26.843898010187118], walls: [0.062491187561917, 0.07594224970907054, 0.08528042770782818]}
    Starting value of the cost: 142.05221871073422
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.09707274] 8.386141954198166
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 2 and 2
    enlarging the TR radius to 0.2
    k: 0 - j 4 - Cost Functional: 113.12019603403996 - mu: {doors: [0.05, 0.07165965759763548], heaters: [41.3617810419687, 69.38608805942935, 95.24934022880507, 53.108086570853885, 51.78677857986843, 11.154448723073822, 24.689931699023266], walls: [0.025, 0.025, 0.1]}
    First order critical condition: 8.359383634977865
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.19485449] 5.436167975475768
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 3 and 3
    enlarging the TR radius to 0.4
    k: 1 - j 4 - Cost Functional: 52.421603857431364 - mu: {doors: [0.05, 0.05], heaters: [34.88613223893655, 62.961312737330125, 90.1633066155099, 50.72682423638229, 55.974891139117915, 19.91458614202501, 17.569650053273897], walls: [0.025, 0.025, 0.1]}
    First order critical condition: 5.435168320471611
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.39903053] 2.701587613005128
    checked sufficient condition
    enlarging the TR radius to 0.8
    Skip_enrichment_step is false, therefore starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 4 and 4
    k: 2 - j 3 - Cost Functional: 17.97344017207331 - mu: {doors: [0.05, 0.05], heaters: [29.933400540210265, 50.73487721387974, 76.66933501212476, 47.37442946711639, 54.10549819580599, 31.060338232313825, 16.777019160578313], walls: [0.026197497656193284, 0.03546662240938889, 0.1]}
    First order critical condition: 2.7063796812009207
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.78366649] 0.9922678308402169
    checked sufficient condition
    enlarging the TR radius to 1.6
    Skip_enrichment_step is false, therefore starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 5 and 5
    k: 3 - j 6 - Cost Functional: 7.550758090439643 - mu: {doors: [0.05, 0.05], heaters: [25.681703313322114, 35.40552760950831, 61.00629445313868, 48.24214443460455, 52.9034095564115, 43.70280954182427, 20.197821013914773], walls: [0.05959950378313627, 0.1, 0.1]}
    First order critical condition: 0.9929574402461987
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    CG truncated at iteration: 0 with residual: 0.54148, because p_k is not a descent direction
    Choosing the gradient as direction
    Using CG for the linear system
    CG truncated at iteration: 1 with residual: 0.05270, because p_k is not a descent direction
    Using CG for the linear system
    CG truncated at iteration: 1 with residual: 0.22345, because p_k is not a descent direction
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.010454493133116548, J_diff = 0.0005138220029902442, FOC = 3.552713678800501e-15
    relative differences mu 0.010454493133116548 and J 0.0005138220029902442
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 6 and 6
    enlarging the TR radius to 3.2
    k: 4 - j 7 - Cost Functional: 5.814067152367443 - mu: {doors: [0.05, 0.05], heaters: [22.417563101082013, 23.553481111884583, 48.785797686897695, 49.181429460254826, 51.97399026564293, 54.29607302982171, 23.52971265562549], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 0.0036581914188549424
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.004752155974444116, J_diff = 1.615701115372057e-05, FOC = 0.0
    relative differences mu 0.004752155974444116 and J 1.615701115372057e-05
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 7 and 7
    enlarging the TR radius to 6.4
    k: 5 - j 2 - Cost Functional: 5.81396524121601 - mu: {doors: [0.05, 0.05], heaters: [22.38297178919087, 23.393225184830648, 48.70727582517131, 49.388585310239705, 52.370127537440446, 54.09655070347377, 23.523731521196268], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 5.877394777836345e-05
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.00023137428403711196, J_diff = 3.026338365582005e-08, FOC = 3.889218348125276e-11
    relative differences mu 0.00023137428403711196 and J 3.026338365582005e-08
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 8 and 8
    enlarging the TR radius to 12.8
    k: 6 - j 2 - Cost Functional: 5.813965062387297 - mu: {doors: [0.05, 0.05], heaters: [22.382540249497715, 23.396509836205237, 48.70345782678151, 49.37420763676373, 52.362755502301376, 54.11554819476222, 23.52381071513161], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 5.459640765978054e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.0, J_diff = 0.0, FOC = 9.93085844921655e-08
    relative differences mu 0.0 and J 0.0
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 9 and 9
    enlarging the TR radius to 25.6
    k: 7 - j 6 - Cost Functional: 5.813965062385478 - mu: {doors: [0.05, 0.05], heaters: [22.38254561408377, 23.396505689849405, 48.70347846121417, 49.37422338503675, 52.362729715619565, 54.11555978719734, 23.523802914232395], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 1.0059844153500897e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.0, J_diff = 0.0, FOC = 9.516489610666632e-08
    relative differences mu 0.0 and J 0.0

    Stopping criteria fulfilled: FOC condition: 9.64746170192378e-08
    k: 8 - j 3 - Cost Functional: 5.813965062881973 - mu: {doors: [0.05, 0.05], heaters: [22.382545610297953, 23.396505685212503, 48.7034784593056, 49.374223383003525, 52.36272971268731, 54.115559784827695, 23.523802904423704], walls: [0.025, 0.025, 0.025]}
    Computing only the initial FOC

```python
if 'Method_4_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4bis_actual,mu_error_4bis_actual)
```

#### Directional Taylor Basis (Method 5)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=True)
opt_fom = opt_fom.with_(adjoint_approach=True)

if 'Method_5' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                prepare_for_hessian=True,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'AdaptiveTaylor_Newton',  'iterative_solver': iterative_solver,
                 'JFE_start': J_start}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_5, times_5, Js_5, FOC_5, data_5 = TR_algorithm(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_5_actual, J_error_5_actual, mu_error_5_actual, FOC_5_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_5, Js_5, times_5, tictoc, FOC_5)
```

    Starting with two bases. Primal and dual have length 1 and 1
    building simple coercive primal reductor...
    building simple coercive dual reductor...
    starting parameter {doors: [0.06144624340609358, 0.1669878188360172], heaters: [43.84092314408935, 72.34651778309413, 97.79895119966027, 53.849587041043364, 50.11204636599379, 7.205113335976154, 26.843898010187118], walls: [0.062491187561917, 0.07594224970907054, 0.08528042770782818]}
    Starting value of the cost: 142.05221871073422
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.09707274] 8.386141954198166
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Length of Bases are 3 and 3
    k: 0 - j 4 - Cost Functional: 113.12019599231144 - mu: {doors: [0.05, 0.07165965759763548], heaters: [41.3617810419687, 69.38608805942935, 95.24934022880507, 53.108086570853885, 51.78677857986843, 11.154448723073822, 24.689931699023266], walls: [0.025, 0.025, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 8.359383634977865
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.09837037] 4.539329874871149
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Length of Bases are 5 and 5
    k: 1 - j 4 - Cost Functional: 38.40411912166633 - mu: {doors: [0.05, 0.05], heaters: [34.38256027687893, 57.37073998029392, 83.3840524459038, 49.65645194586021, 52.315216585388484, 22.194921416217756, 19.402225233769467], walls: [0.025, 0.025, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.539559577934062
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    CG truncated at iteration: 13 with residual: 1.32765, because p_k is not a descent direction
    Using CG for the linear system
    CG truncated at iteration: 13 with residual: 1.35231, because p_k is not a descent direction
    Maximum iteration for Armijo rule reached
    boundary criterium of the TR satisfied, so stopping the sub-problem 0.1 2.2428336434277294
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Length of Bases are 7 and 7
    k: 2 - j 5 - Cost Functional: 14.327771310067078 - mu: {doors: [0.05, 0.05], heaters: [30.114874821022962, 47.409741773884086, 73.74491521526073, 48.68479061902178, 51.546158043302384, 29.841142344123625, 17.982258935200132], walls: [0.03672058185400817, 0.048868168073713136, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 2.2426798259858143
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.09526129] 0.6813897389400295
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Length of Bases are 9 and 9
    k: 3 - j 4 - Cost Functional: 6.739539927176807 - mu: {doors: [0.05, 0.05], heaters: [25.770913175062816, 34.112428744936324, 59.88176903606544, 48.5935310663026, 51.490152115680075, 42.73233035932198, 20.422421001056893], walls: [0.05393579933636174, 0.1, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.6813618831685768
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    CG truncated at iteration: 1 with residual: 0.23363, because p_k is not a descent direction
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.011022457035383465, J_diff = 0.0011347586110180832, FOC = 8.702335715267317e-15
    relative differences mu 0.011022457035383465 and J 0.0011347586110180832
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Length of Bases are 11 and 11
    k: 4 - j 5 - Cost Functional: 5.814190952796476 - mu: {doors: [0.05, 0.05], heaters: [22.40398774694965, 23.389677720720027, 48.676834379216004, 49.392197453803654, 52.368949986849735, 54.09764614085354, 23.522758899702787], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 5.448583097595098e-05
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.0004054467582519142, J_diff = 1.501860298366099e-07, FOC = 2.2469334198890888e-14
    relative differences mu 0.0004054467582519142 and J 1.501860298366099e-07
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Length of Bases are 13 and 13
    k: 5 - j 2 - Cost Functional: 5.813965063448125 - mu: {doors: [0.05, 0.05], heaters: [22.382300521263065, 23.396707593322063, 48.70383326441332, 49.37389091736957, 52.361842505204436, 54.116459460906356, 23.523800883937948], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 9.373905332326071e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 1.275242931007516e-05, J_diff = 9.766721571244475e-11, FOC = 1.8613803471413335e-11
    relative differences mu 1.275242931007516e-05 and J 9.766721571244475e-11
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Length of Bases are 15 and 15
    k: 6 - j 2 - Cost Functional: 5.813965062889002 - mu: {doors: [0.05, 0.05], heaters: [22.382542667233334, 23.39650219680923, 48.703486386237785, 49.374229154104206, 52.362739191003314, 54.115550419022064, 23.52380227638022], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 3.2102048359099294e-08
    *******************************

    Stopping criteria fulfilled: FOC condition: 3.2102048359099294e-08
    Computing only the initial FOC

```python
if 'Method_5' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5_actual,mu_error_5_actual)
```

#### Directional Taylor Basis with optional enrichment (Method 5.a)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=True)
opt_fom = opt_fom.with_(adjoint_approach=True)

if 'Method_5_optional' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                prepare_for_hessian=True,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,  'iterative_solver': iterative_solver,
                 'tau_g': tau_g, 'tau_grad': tau_grad, 'beta_3': beta_3,
                 'minimum_radius': minimum_radius, 'maximum_radius': maximum_radius,
                 'opt_method': 'AdaptiveTaylor_Newton',
                 'JFE_start': J_start}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_5bis, times_5bis, Js_5bis, FOC_5bis, data_5bis = TR_algorithm_with_optional_enrichment(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_5bis_actual, J_error_5bis_actual, mu_error_5bis_actual, FOC_5bis_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_5bis, Js_5bis, times_5bis, tictoc, FOC_5bis)
```

    Starting with two bases. Primal and dual have length 1 and 1
    building simple coercive primal reductor...
    building simple coercive dual reductor...
    starting parameter {doors: [0.06144624340609358, 0.1669878188360172], heaters: [43.84092314408935, 72.34651778309413, 97.79895119966027, 53.849587041043364, 50.11204636599379, 7.205113335976154, 26.843898010187118], walls: [0.062491187561917, 0.07594224970907054, 0.08528042770782818]}
    Starting value of the cost: 142.05221871073422
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.09707274] 8.386141954198166
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Length of Bases are 3 and 3
    enlarging the TR radius to 0.2
    k: 0 - j 4 - Cost Functional: 113.12019603403996 - mu: {doors: [0.05, 0.07165965759763548], heaters: [41.3617810419687, 69.38608805942935, 95.24934022880507, 53.108086570853885, 51.78677857986843, 11.154448723073822, 24.689931699023266], walls: [0.025, 0.025, 0.1]}
    First order critical condition: 8.359383634977865
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.19830155] 4.253983620316258
    checked sufficient condition
    enlarging the TR radius to 0.4
    Skip_enrichment_step is false, therefore starting enrichment
    building simple coercive dual reductor...
    Length of Bases are 5 and 5
    k: 1 - j 3 - Cost Functional: 34.35437853645641 - mu: {doors: [0.05, 0.05], heaters: [33.26353161603027, 54.886846748808736, 80.50523110747785, 49.14725420999213, 52.95983563934495, 25.810353604566235, 19.670180546780006], walls: [0.025, 0.025, 0.1]}
    First order critical condition: 4.254577351752512
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.39958281] 2.056678969153937
    checked sufficient condition
    enlarging the TR radius to 0.8
    Skip_enrichment_step is false, therefore starting enrichment
    building simple coercive dual reductor...
    Length of Bases are 7 and 7
    k: 2 - j 4 - Cost Functional: 12.917885268362568 - mu: {doors: [0.05, 0.05], heaters: [28.895372171345585, 44.93976172099176, 71.0028280774457, 48.361590366043366, 52.64405263301422, 34.03052215282422, 18.35048033190267], walls: [0.047661287290188974, 0.1, 0.1]}
    First order critical condition: 2.0556554662700344
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    boundary criterium of the TR satisfied, so stopping the sub-problem [0.78841326] 0.4958002095351855
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Length of Bases are 9 and 9
    enlarging the TR radius to 1.6
    k: 3 - j 4 - Cost Functional: 6.3209111390178805 - mu: {doors: [0.05, 0.05], heaters: [24.791658315381184, 31.16037442566547, 56.76005395334332, 48.7586154567401, 52.06267696094298, 46.26771732015814, 21.249850204645362], walls: [0.1, 0.1, 0.1]}
    First order critical condition: 0.49649365773773557
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Using CG for the linear system
    Using CG for the linear system
    CG truncated at iteration: 1 with residual: 0.24021, because p_k is not a descent direction
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.010595532782200528, J_diff = 0.0011515851258599658, FOC = 0.0
    relative differences mu 0.010595532782200528 and J 0.0011515851258599658
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Length of Bases are 11 and 11
    enlarging the TR radius to 3.2
    k: 4 - j 5 - Cost Functional: 5.813968598826705 - mu: {doors: [0.05, 0.05], heaters: [22.414681914251126, 23.39010925067939, 48.71662300239289, 49.43616557222926, 52.293233412521694, 54.0941252205982, 23.51745511560683], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 0.0001529596522237895
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 0.0009289602256799379, J_diff = 6.066378929551331e-07, FOC = 0.0
    relative differences mu 0.0009289602256799379 and J 6.066378929551331e-07
    Sufficient decrease condition fulfilled
    Enriching the RB model
    building simple coercive dual reductor...
    Length of Bases are 13 and 13
    enlarging the TR radius to 6.4
    k: 5 - j 2 - Cost Functional: 5.813965062730176 - mu: {doors: [0.05, 0.05], heaters: [22.38229724034617, 23.396746143504707, 48.703724904526716, 49.373695613485204, 52.36220702536381, 54.11601083541569, 23.5238837018478], walls: [0.025, 0.025, 0.025]}
    First order critical condition: 1.9954805368919493e-06
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the Newton method
    Using CG for the linear system
    Subproblem converged: mu_diff = 9.076522420084405e-06, J_diff = 6.018567785679375e-11, FOC = 1.9888119098057417e-11
    relative differences mu 9.076522420084405e-06 and J 6.018567785679375e-11

    Stopping criteria fulfilled: FOC condition: 2.2494923335864352e-08
    k: 6 - j 2 - Cost Functional: 5.813965062878147 - mu: {doors: [0.05, 0.05], heaters: [22.38255198750563, 23.39649992696152, 48.7034782678235, 49.37423538650612, 52.362747121633745, 54.115541474905825, 23.523799702888795], walls: [0.025, 0.025, 0.025]}
    Computing only the initial FOC

```python
if 'Method_5_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5bis_actual,mu_error_5bis_actual)
```

### Aggregated

#### excluding directional sensitivities (Method 6) -- (b)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=False)
opt_fom = opt_fom.with_(adjoint_approach=False)

if 'Method_6' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                prepare_for_hessian=True,
                                                unique_basis=True,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'NewtonMethod',  'iterative_solver': iterative_solver}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_6, times_6, Js_6, FOC_6, data_6 = TR_algorithm(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_6_actual, J_error_6_actual, mu_error_6_actual, FOC_6_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_6, Js_6, times_6, tictoc, FOC_6)
```

```python
if 'Method_6' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_6_actual, mu_error_6_actual)
```

#### including directional sensitivities (Method 7) -- (d)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=False)
opt_fom = opt_fom.with_(adjoint_approach=False)

if 'Method_7' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                prepare_for_hessian=True,
                                                unique_basis=True,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'estimate_tolerance': estimate_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'AdaptiveTaylor_Newton',  'iterative_solver': iterative_solver}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_7, times_7, Js_7, FOC_7, data_7 = TR_algorithm(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_7_actual, J_error_7_actual, mu_error_7_actual, FOC_7_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_7, Js_7, times_7, tictoc, FOC_7)
```

```python
if 'Method_7' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_7_actual,mu_error_7_actual)
```

## NCD corrected BFGS Method (KMSOV'20)

```python
tic = time.time()
params = [mu]

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=True)
opt_fom = opt_fom.with_(adjoint_approach=True)

if 'Method_8' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(
        opt_fom, params, build_sensitivities=False)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                RBbasis, dual_RBbasis,
                                                opt_product=opt_fom.opt_product,
                                                coercivity_estimator=ce,
                                                reductor_type=reductor_type, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()

    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'BFGSMethod',  'iterative_solver': iterative_solver}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_8, times_8, Js_8, FOC_8, data_8 = TR_algorithm(opt_rom, pdeopt_reductor, parameter_space,
                                                TR_parameters, extension_params)

    times_full_8_actual, J_error_8_actual, mu_error_8_actual, FOC_8_actual = compute_errors(
        opt_fom, parameter_space, J_start, J_opt, mu, mu_opt_as_array, mus_8, Js_8, times_8, tictoc, FOC_8)
```

    Starting with two bases. Primal and dual have length 1 and 1
    building simple coercive primal reductor...
    building simple coercive dual reductor...
    starting parameter {doors: [0.06144624340609358, 0.1669878188360172], heaters: [43.84092314408935, 72.34651778309413, 97.79895119966027, 53.849587041043364, 50.11204636599379, 7.205113335976154, 26.843898010187118], walls: [0.062491187561917, 0.07594224970907054, 0.08528042770782818]}
    Starting value of the cost: 142.05221871073422
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 2 and 2
    k: 0 - j 2 - Cost Functional: 114.45407351670019 - mu: {doors: [0.05, 0.057542402967186886], heaters: [42.78954241131653, 71.23303262313476, 97.22952227306698, 53.27985164813631, 49.506049318948016, 6.590006186596227, 24.67196708006731], walls: [0.025, 0.025, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 8.39933557050463
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 3 and 3
    k: 1 - j 4 - Cost Functional: 47.51306558924125 - mu: {doors: [0.05, 0.05], heaters: [39.214520419481346, 67.43416976165338, 95.2805335701969, 51.32700524652157, 47.41391676560445, 4.448727295439071, 17.253301318838986], walls: [0.025, 0.025, 0.1]}
    computing the fom gradient since the model was enriched
    First order critical condition: 5.064772769638789
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 4 and 4
    k: 2 - j 7 - Cost Functional: 9.235760438391742 - mu: {doors: [0.05, 0.05], heaters: [33.67020503380064, 61.40817331097209, 92.23173915313825, 48.328499767541594, 44.209970877040405, 1.224179135072411, 6.0414351303941976], walls: [0.04822085600622893, 0.03602027459775794, 0.09645118475811346]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.1492130794780555
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 5 and 5
    k: 3 - j 1 - Cost Functional: 9.228003369576921 - mu: {doors: [0.05422255208892537, 0.05], heaters: [33.667308482560685, 61.398556962060326, 92.22893727586872, 48.32847704215824, 44.210438259271825, 1.22756831551033, 6.050044718732929], walls: [0.06922504250193089, 0.06898718182921976, 0.09100536031808307]}
    Shrinking the TR radius to: 0.05 because Jcp 9.227744074353902 and J_kp1 9.228003369576921
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 5 and 5
    k: 3 - j 4 - Cost Functional: 9.2261357783174 - mu: {doors: [0.05, 0.05], heaters: [33.65901226013841, 61.377983802274834, 92.2221543059777, 48.32665070480215, 44.20927721162202, 1.2315113126726176, 6.058799970172423], walls: [0.0657647546945632, 0.059267856716550535, 0.08425428845681641]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.1296018489751725
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 6 and 6
    k: 4 - j 4 - Cost Functional: 8.956285803139476 - mu: {doors: [0.05, 0.05], heaters: [32.9419029399073, 59.45455782090619, 91.61275634871608, 48.21211301825747, 44.17389572948895, 1.7113551942568035, 7.170910556450257], walls: [0.1, 0.1, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.10984147175933355
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 7 and 7
    k: 5 - j 3 - Cost Functional: 8.70557934953586 - mu: {doors: [0.05, 0.05], heaters: [32.15670458692938, 56.34706334598221, 90.71558906719875, 48.31489620805253, 44.453798232659196, 3.033844276191076, 10.494473316965998], walls: [0.09842133348988213, 0.09783217183955414, 0.027563336683212456]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.41353954223151296
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 8 and 8
    k: 6 - j 10 - Cost Functional: 6.993122170469121 - mu: {doors: [0.05, 0.05], heaters: [23.321592872214037, 29.390562498114317, 81.51843305056897, 47.52109657971681, 45.052991068434764, 12.362915861176749, 28.388952904327223], walls: [0.07458838343098202, 0.025, 0.04054416696316503]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.5016150695425821
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: -8.136729642971634e-07
    Reset direction to - gradient
    Curvature condition: -1.3390718882382175e-06
    Reset direction to - gradient
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 9 and 9
    k: 7 - j 7 - Cost Functional: 6.610650536017161 - mu: {doors: [0.05, 0.05], heaters: [22.797175169350233, 28.78056050510612, 81.13586027596982, 47.242192192906074, 44.77305252762322, 12.182109010497452, 27.20315090055979], walls: [0.07217235150347445, 0.1, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.03629596418441001
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: -0.0007283689781202494
    Reset direction to - gradient
    Curvature condition: -0.0001585660363539747
    Reset direction to - gradient
    Curvature condition: -2.422048826420664e-05
    Reset direction to - gradient
    Curvature condition: -8.399022224532402e-06
    Reset direction to - gradient
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 10 and 10
    k: 8 - j 4 - Cost Functional: 6.608400914175661 - mu: {doors: [0.05, 0.05], heaters: [22.800936203739944, 28.772639924678643, 81.1191453160296, 47.24570677258488, 44.780094876787025, 12.209743276629421, 27.19542622879611], walls: [0.040974190528857146, 0.1, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.03427250670308471
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: -0.002296859300078467
    Reset direction to - gradient
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 11 and 11
    k: 9 - j 9 - Cost Functional: 6.466350747103501 - mu: {doors: [0.05, 0.05], heaters: [23.38901501865395, 27.29361468688337, 78.0726879190463, 47.87488843021095, 46.07542001377582, 17.242478452832838, 25.797309165031873], walls: [0.025, 0.1, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.1627246598800718
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 12 and 12
    k: 10 - j 9 - Cost Functional: 5.978030969381397 - mu: {doors: [0.05, 0.05], heaters: [24.5591225856949, 23.044021001272846, 65.73098801802368, 50.34258864242191, 51.12869830805166, 37.275615984372315, 23.26293818599633], walls: [0.025, 0.08782130538524255, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.07047242594407369
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: -0.023792660196883437
    Reset direction to - gradient
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 13 and 13
    k: 11 - j 13 - Cost Functional: 5.933647257358704 - mu: {doors: [0.05, 0.05180407060129789], heaters: [23.316888245718538, 23.161983388930057, 63.418043649400296, 50.158250285368176, 51.28939486154141, 39.619232267980166, 23.433608885661442], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.076595688925621
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    boundary criterium of the TR satisfied, so stopping the sub-problem solver
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 14 and 14
    k: 12 - j 49 - Cost Functional: 5.820802581904747 - mu: {doors: [0.05, 0.05], heaters: [20.302268455956483, 23.993296777291437, 51.042882746888964, 49.620839624187724, 52.56523147457021, 52.74565411279851, 23.87611373840908], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 0.005764428575472224
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 2.1695381996760184e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 15 and 15
    k: 13 - j 361 - Cost Functional: 5.813965100939438 - mu: {doors: [0.05, 0.05], heaters: [22.382935388883954, 23.396489696589505, 48.69694853332114, 49.38248175278316, 52.36579425988762, 54.110081213352956, 23.52387362979278], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 6.644252893086533e-06
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 3.941449634157848e-07, mu_diff = 1.5614980450787851e-15, J_diff = 0.0
    relative differences mu 1.5614980450787851e-15 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 16 and 16
    k: 14 - j 134 - Cost Functional: 5.813965062938548 - mu: {doors: [0.05, 0.05], heaters: [22.38248661060615, 23.396502388916485, 48.70357655001728, 49.37427500129998, 52.362798302224384, 54.11584170194356, 23.523692702326017], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 3.9331506659436866e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 7.956445309362526e-07, mu_diff = 7.917109797490563e-17, J_diff = 3.055327681145635e-16
    relative differences mu 7.917109797490563e-17 and J 3.055327681145635e-16
    checked sufficient condition, starting enrichment
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 17 and 17
    k: 15 - j 7 - Cost Functional: 5.813965062936168 - mu: {doors: [0.05, 0.05], heaters: [22.382485841027016, 23.39649930849508, 48.70357403519678, 49.37427287096308, 52.36279572100181, 54.11583711490311, 23.52369438132341], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 7.944539815587382e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 6.43206334138827e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 18 and 18
    k: 16 - j 6 - Cost Functional: 5.81396506293779 - mu: {doors: [0.05, 0.05], heaters: [22.382487241754284, 23.396500111616703, 48.70357416927901, 49.37427313460783, 52.362795917212296, 54.11583674727705, 23.523698264462162], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 6.443321806665692e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 6.370703174283562e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 19 and 19
    k: 17 - j 7 - Cost Functional: 5.813965062932375 - mu: {doors: [0.05, 0.05], heaters: [22.38248617475625, 23.396498201899117, 48.70357283210168, 49.374271929764966, 52.36279451725211, 54.11583460368765, 23.52369709750962], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 6.359471766145699e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.5417265329618957e-07, mu_diff = 6.395919428807219e-13, J_diff = 0.0
    relative differences mu 6.395919428807219e-13 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 20 and 20
    k: 18 - j 3 - Cost Functional: 5.813965062934875 - mu: {doors: [0.05, 0.05], heaters: [22.382486443605384, 23.396498330014882, 48.70357283107492, 49.37427195855531, 52.362794527218576, 54.11583447922965, 23.523697873498524], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.5330089534689094e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 5.540689560455115e-07, mu_diff = 1.4989085063638737e-12, J_diff = 0.0
    relative differences mu 1.4989085063638737e-12 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 21 and 21
    k: 19 - j 6 - Cost Functional: 5.8139650629337485 - mu: {doors: [0.05, 0.05], heaters: [22.382487331028237, 23.39649839223522, 48.703572477197355, 49.374271769091344, 52.36279420758129, 54.11583338471027, 23.52370091673268], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 5.551293468450587e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.6161698355671917e-07, mu_diff = 3.7414802877718433e-16, J_diff = 0.0
    relative differences mu 3.7414802877718433e-16 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 20 - j 7 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 21 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 22 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 23 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 24 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 25 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 26 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 27 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 28 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 29 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 22
    k: 30 - j 1 - Cost Functional: 5.8139650629296895 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616123989368788e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 22 and 23
    k: 31 - j 1 - Cost Functional: 5.813965062933002 - mu: {doors: [0.05, 0.05], heaters: [22.382486596122146, 23.396497027319448, 48.70357149417669, 49.374270892206134, 52.36279318492598, 54.115831789422764, 23.523700152994575], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.607013772699369e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.616003067241028e-07, mu_diff = 1.0252960455818208e-12, J_diff = 0.0
    relative differences mu 1.0252960455818208e-12 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 32 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.523700153096524], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 3.2321466855410755e-17, J_diff = 0.0
    relative differences mu 3.2321466855410755e-17 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 33 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.523700153096527], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: -5.40674174994047e-31
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 3.2321466855410755e-17, J_diff = 0.0
    relative differences mu 3.2321466855410755e-17 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 34 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309653], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 35 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309653], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 36 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309653], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 37 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309653], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 24
    k: 38 - j 1 - Cost Functional: 5.813965062935075 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309653], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.616040939270499e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 39 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153534, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309653], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.615958607897774e-07, mu_diff = 7.227299702120482e-17, J_diff = 0.0
    relative differences mu 7.227299702120482e-17 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 40 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.523700153096538], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 3.2321466855410755e-17, J_diff = 0.0
    relative differences mu 3.2321466855410755e-17 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 41 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309654], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Subproblem converged: FOC = 4.6159585668899715e-07, mu_diff = 3.2321466855410755e-17, J_diff = 0.0
    relative differences mu 3.2321466855410755e-17 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 42 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.523700153096545], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: -2.0518094026371317e-31
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 3.2321466855410755e-17, J_diff = 0.0
    relative differences mu 3.2321466855410755e-17 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 43 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 44 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 45 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 46 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 47 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 48 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 25
    k: 49 - j 1 - Cost Functional: 5.81396506293059 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.6159585767848214e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 50 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 51 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 52 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 53 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 54 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 55 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 56 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 57 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 26
    k: 58 - j 1 - Cost Functional: 5.813965062930105 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

    ___ starting subproblem
    Computing the approximate Cauchy point and then start the BFGS method
    Curvature condition: 0.0
    Reset direction to - gradient
    Subproblem converged: FOC = 4.615949443175215e-07, mu_diff = 0.0, J_diff = 0.0
    relative differences mu 0.0 and J 0.0
    enriching to check the sufficient decrease condition
    building simple coercive dual reductor...
    Enrichment completed... length of Bases are 23 and 27
    k: 59 - j 1 - Cost Functional: 5.813965062933441 - mu: {doors: [0.05, 0.05], heaters: [22.382486596153537, 23.39649702732504, 48.70357149416697, 49.374270892201714, 52.362793184917514, 54.11583178938952, 23.52370015309655], walls: [0.025, 0.025, 0.025]}
    computing the fom gradient since the model was enriched
    First order critical condition: 4.606831743910357e-07
    *******************************

     WARNING: Maximum number of iteration for the TR algorithm reached
    Computing only the initial FOC

```python
if 'Method_8' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_8_actual,mu_error_8_actual)
```

## Qian-Grepl-Method

```python
#Qian-Grepl

# make sure to use the correct config
opt_fom = opt_fom.with_(use_corrected_functional=False)
opt_fom = opt_fom.with_(adjoint_approach=False)
tic = time.time()
params = [mu]

if 'Method_9' in optimization_methods or 'All' in optimization_methods:

    RBbasis, dual_RBbasis = build_initial_basis(opt_fom, params)

    pdeopt_reductor = QuadraticPdeoptStationaryCoerciveReductor(opt_fom,
                                                                RBbasis, dual_RBbasis,
                                                                opt_product=opt_fom.opt_product,
                                                                prepare_for_hessian=False,
                                                                prepare_for_gradient_estimate=True,
                                                                coercivity_estimator=ce, mu_bar=mu_bar)

    opt_rom = pdeopt_reductor.reduce()
    tictoc = time.time() - tic

    TR_parameters = {'Qian-Grepl': True,
                 'Qian-Grepl_subproblem': Qian_Grepl_subproblem, 'beta': beta,
                 'safety_tolerance': safety_tol,
                 'radius': radius, 'FOC_tolerance': FOC_tolerance,
                 'sub_tolerance': sub_tolerance,
                 'max_iterations': max_it, 'max_iterations_subproblem': max_it_sub,
                 'max_iterations_armijo': max_it_arm,
                 'initial_step_armijo': init_step_armijo,
                 'armijo_alpha': armijo_alpha,
                 'epsilon_i': epsilon_i,
                 'control_mu': control_mu,
                 'starting_parameter': mu,
                 'opt_method': 'BFGSMethod',  'iterative_solver': iterative_solver}

    extension_params = {'Enlarge_radius': False, 'timings': True,
                        'opt_fom': opt_fom, 'return_data_dict': True}

    mus_9, times_9, Js_9, FOC_9, data_9 = TR_algorithm(
        opt_rom, pdeopt_reductor, parameter_space, TR_parameters, extension_params)

    times_full_9_actual, J_error_9_actual, FOC_9_actual, mu_error_9_actual = compute_actual_errors(
        opt_fom, parameter_space, J_start, J_opt, mus_9, times_9, tictoc, mu, mu_opt_as_array)

```

```python
if 'Method_9' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_9_actual,J_error_9_actual)
```

# Results

```python
print(mu_opt)
u = opt_fom.solve(mu_opt)
opt_fom.visualize(u)
```

    {doors: [0.05, 0.05], heaters: [22.38254711947, 23.39650455721, 48.70348426512, 49.37422778852, 52.36272251326, 54.11556305764, 23.52380079299], walls: [0.025, 0.025, 0.025]}

    Output()

## Plot results

### J error

```python
timings_figure = plt.figure()
if 'SH' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_SH,J_error_SH,'o-', label='SH Method')
if 'Method_4' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4_actual,J_error_4_actual,'o-', label='4. Newton NCD-corrected Lag.')
if 'Method_5' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5_actual,J_error_5_actual,'o-', label='5. Newton NCD-corrected Dir.Tay.')
if 'Method_4_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4bis_actual,J_error_4bis_actual,'o-', label='4.a Newton NCD-corrected Lag. optional')
if 'Method_5_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5bis_actual,J_error_5bis_actual,'o-', label='5.a Newton NCD-corrected Dir.Tay. optional')
if 'Method_6' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_6_actual,J_error_6_actual,'o-', label='6. Newton Aggr.')
if 'Method_7' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_7_actual,J_error_7_actual,'o-', label='7. Newton Aggr. Dir.Tay')
if 'Method_8' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_8_actual,J_error_8_actual,'o-', label='8. NCD-corrected BFGS')
if 'Method_9' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_9_actual,J_error_9_actual,'o-', label='9. Qian-Grepl')

plt.xlabel('time in seconds [s]')
plt.ylabel('True optimization error of the output functional')
plt.grid()
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
#timings_figure.savefig('compare_FEM_TR', format='pdf', bbox_inches="tight")
```

### Plot FOC

```python
timings_figure = plt.figure()

if 'SH' in optimization_methods or 'FOM' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_SH, FOC_SH,'o-', label='SH Method')
if 'Method_4' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4_actual, FOC_4_actual,'o-', label='4. Newton NCD-corrected Lag.')
if 'Method_5' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5_actual, FOC_5_actual,'o-', label='5. Newton NCD-corrected Dir.Tay.')
if 'Method_4_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4bis_actual, FOC_4bis_actual,'o-', label='4.a Newton NCD-corrected Lag.')
if 'Method_5_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5bis_actual, FOC_5bis_actual,'o-', label='5.a Newton NCD-corrected Dir.Tay.')
if 'Method_6' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_6_actual, FOC_6_actual,'o-', label='6. Newton Aggr.')
if 'Method_7' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_7_actual, FOC_7_actual,'o-', label='7. Newton Aggr. Dir.Tay')
if 'Method_8' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_8_actual, FOC_8_actual,'o-', label='8. NCD-corrected BFGS')
if 'Method_9' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_9_actual, FOC_9_actual,'o-', label='9. Qian-Grepl')

plt.xlabel('time in seconds [s]')
plt.ylabel('First-order critical condition')
# plt.xlim([-1,30])
plt.grid()
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
#timings_figure.savefig('compare_FEM_TR', format='pdf', bbox_inches="tight")
```

### Plot Mu error

```python
if 'SH' in optimization_methods or 'FOM' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_SH,mu_error_SH,'o-', label='SH Method')
if 'Method_4' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4_actual,mu_error_4_actual,'o-', label='4. Newton NCD-corrected Lag.')
if 'Method_5' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5_actual,mu_error_5_actual,'o-', label='5. Newton NCD-corrected Dir.Tay.')
if 'Method_4_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_4bis_actual,mu_error_4bis_actual,'o-', label='4.a Newton NCD-corrected Lag.')
if 'Method_5_optional' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_5bis_actual,mu_error_5bis_actual,'o-', label='5.a Newton NCD-corrected Dir.Tay.')
if 'Method_6' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_6_actual,mu_error_6_actual,'o-', label='6. Newton Aggr.')
if 'Method_7' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_7_actual,mu_error_7_actual,'o-', label='7. Newton Aggr. Dir.Tay')
if 'Method_8' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_8_actual,mu_error_8_actual,'o-', label='8. NCD-corrected BFGS')
if 'Method_9' in optimization_methods or 'All' in optimization_methods:
    plt.semilogy(times_full_9_actual,mu_error_9_actual,'o-', label='9. Qian-Grepl')

plt.xlabel('time in seconds [s]')
plt.ylabel('Mu error')
#plt.xlim([-1,100])
plt.grid()
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
#timings_figure.savefig('compare_FEM_TR', format='pdf', bbox_inches="tight")
```

### Save the data

```python
#save data
from pdeopt.tools import save_data

directory = f'results/Starter{seed}'

if 'SH' in optimization_methods or 'FOM' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_SH, J_error_SH, 1, mu_error=mu_error_SH, FOC=FOC_SH)
if 'Method_4' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_4_actual, J_error_4_actual, 4, mu_error=mu_error_4_actual, FOC=FOC_4_actual, additional_data=data_4)
if 'Method_5' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_5_actual, J_error_5_actual, 5, mu_error=mu_error_5_actual, FOC=FOC_5_actual, additional_data=data_5)
if 'Method_4_optional' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_4bis_actual, J_error_4bis_actual, 2, mu_error=mu_error_4bis_actual, FOC=FOC_4bis_actual, additional_data=data_4bis)
if 'Method_5_optional' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_5bis_actual, J_error_5bis_actual, 3, mu_error=mu_error_5bis_actual, FOC=FOC_5bis_actual, additional_data=data_5bis)
if 'Method_6' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_6_actual, J_error_6_actual, 6, mu_error=mu_error_6_actual, FOC=FOC_6_actual, additional_data=data_6)
if 'Method_7' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_7_actual, J_error_7_actual, 7, mu_error=mu_error_7_actual, FOC=FOC_7_actual, additional_data=data_7)
if 'Method_8' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_8_actual, J_error_8_actual, 8, mu_error=mu_error_8_actual, FOC=FOC_8_actual, additional_data=data_8)
if 'Method_9' in optimization_methods or 'All' in optimization_methods:
    save_data(directory, times_full_9_actual, J_error_9_actual, 9, mu_error=mu_error_9_actual, FOC=FOC_9_actual, additional_data=data_9)
```