Tutorial: Sector-Coupled#
Note
If you have not yet installed PyPSA-Earth, please refer to the Installation section.
In this tutorial, we will show you how to run the sector-coupled model. The sector-coupled model is a model that considers the energy system as a whole, including the electricity, heat, transport, and industry sectors. The sector-coupled model is a global model that can be used to model any region of the Earth. This section explains how to run and analyze the tutorial model.
The sector-coupling code can be run as an overnight/greenfield scenario or myopic scenario. The overnight scenario is a long-term scenario that runs for a year, while the myopic scenario is a short-term scenario that runs for a day.
Overnight Scenarios#
Configuration#
All the configuration for a sector-coupled run are present in the config.default.yaml file.
In particular, the default value for foresight parameter is set to overnight. For the purpose
of this tutorial, test/config.sector.yaml will be used in addition to config.default.yaml
to run the sector-coupled model. That allows you to use a lightweight tutorial datakit
enabled with tutorial: true.
foresight: overnight
Documentation for all options is currently being updated in Configuration.
Scenarios can be defined like for electricity-only studies, but with additional wildcard options.
Note
It is important to set the following flags retrieve_databundle and retrieve_databundle_sector
to false after the first run to prevent unnecessary re-downloads, as the files only need to be downloaded once.
enable:
retrieve_databundle: true
retrieve_databundle_sector: true
scenario:
simpl: [""]
ll: ["copt"]
clusters: [10]
opts: [Co2L-3h]
planning_horizons: # investment years for myopic and perfect; or costs year for overnight
- 2030
sopts:
- "144h"
demand:
- "AB"
For allowed wildcard values, refer to Wildcards.
Execution#
To run the tutorial for the sector-coupled model, you need to activate the pypsa-earth environment. You need to have installed PyPSA-Earth using the instructions provided in the Installation section. Make sure to be in the PyPSA-Earth root directory and run the following command:
Note
It is good practice to perform a dry-run using the option -n, before you commit to a run:
.../pypsa-earth (pypsa-earth) $ snakemake solve_sector_networks -j2 --configfile test/config.sector.yaml -n
.../pypsa-earth (pypsa-earth) $ conda activate pypsa-earth
.../pypsa-earth (pypsa-earth) $ snakemake solve_sector_networks -j2 --configfile test/config.sector.yaml
This covers the retrieval of additional raw data from online resources and preprocessing data about the transport, industry, and heating sectors as well as additional rules about geological storage and sequestration potentials, gas infrastructure, and biomass potentials. The workflow extracts all the data needed to run a model for any country of the world.
This triggers a workflow of multiple preceding jobs that depend on each rule’s inputs and outputs:
![digraph snakemake_dag {
graph[bgcolor=white, margin=0];
node[shape=box, style=rounded, fontname=sans, fontsize=10, penwidth=2];
edge[penwidth=2, color=grey];
0[label = "solve_sector_networks", color = "0.50 0.6 0.85", style="rounded"];
1[label = "solve_sector_network", color = "0.25 0.6 0.85", style="rounded"];
2[label = "add_export", color = "0.08 0.6 0.85", style="rounded"];
3[label = "prepare_ports", color = "0.06 0.6 0.85", style="rounded"];
4[label = "retrieve_cost_data\nyear: 2030", color = "0.05 0.6 0.85", style="rounded"];
5[label = "build_ship_profile\nh2export: 10", color = "0.34 0.6 0.85", style="rounded"];
6[label = "prepare_sector_network", color = "0.28 0.6 0.85", style="rounded"];
7[label = "override_respot\ndiscountrate: 0.071\nsopts: 144h", color = "0.20 0.6 0.85", style="rounded"];
8[label = "prepare_network\nll: copt\nopts: Co2L-4H", color = "0.53 0.6 0.85", style="rounded"];
9[label = "add_extra_components", color = "0.24 0.6 0.85", style="rounded"];
10[label = "cluster_network\nclusters: 6", color = "0.35 0.6 0.85", style="rounded"];
11[label = "simplify_network\nsimpl: ", color = "0.44 0.6 0.85", style="rounded"];
12[label = "add_electricity", color = "0.49 0.6 0.85", style="rounded"];
13[label = "build_renewable_profiles\ntechnology: onwind", color = "0.52 0.6 0.85", style="rounded"];
14[label = "build_natura_raster", color = "0.41 0.6 0.85", style="rounded"];
15[label = "retrieve_databundle_light", color = "0.23 0.6 0.85", style="rounded,dashed"];
16[label = "build_shapes", color = "0.65 0.6 0.85", style="rounded"];
17[label = "build_powerplants", color = "0.12 0.6 0.85", style="rounded"];
18[label = "base_network", color = "0.02 0.6 0.85", style="rounded"];
19[label = "build_osm_network", color = "0.04 0.6 0.85", style="rounded"];
20[label = "clean_osm_data", color = "0.14 0.6 0.85", style="rounded"];
21[label = "download_osm_data", color = "0.40 0.6 0.85", style="rounded"];
22[label = "build_bus_regions", color = "0.66 0.6 0.85", style="rounded"];
23[label = "build_renewable_profiles\ntechnology: offwind-ac", color = "0.52 0.6 0.85", style="rounded"];
24[label = "build_renewable_profiles\ntechnology: offwind-dc", color = "0.52 0.6 0.85", style="rounded"];
25[label = "build_renewable_profiles\ntechnology: solar", color = "0.52 0.6 0.85", style="rounded"];
26[label = "build_renewable_profiles\ntechnology: hydro", color = "0.52 0.6 0.85", style="rounded"];
27[label = "build_demand_profiles", color = "0.39 0.6 0.85", style="rounded"];
28[label = "prepare_energy_totals\ndemand: AB\nplanning_horizons: 2030", color = "0.29 0.6 0.85", style="rounded"];
29[label = "build_base_energy_totals", color = "0.10 0.6 0.85", style="rounded"];
30[label = "prepare_heat_data", color = "0.26 0.6 0.85", style="rounded"];
31[label = "build_clustered_population_layouts", color = "0.17 0.6 0.85", style="rounded"];
32[label = "build_population_layouts\nplanning_horizons: 2030", color = "0.11 0.6 0.85", style="rounded"];
33[label = "prepare_urban_percent", color = "0.13 0.6 0.85", style="rounded"];
34[label = "build_temperature_profiles", color = "0.60 0.6 0.85", style="rounded"];
35[label = "build_cop_profiles", color = "0.47 0.6 0.85", style="rounded"];
36[label = "build_solar_thermal_profiles", color = "0.36 0.6 0.85", style="rounded"];
37[label = "build_heat_demand", color = "0.38 0.6 0.85", style="rounded"];
38[label = "prepare_transport_data", color = "0.46 0.6 0.85", style="rounded"];
39[label = "prepare_transport_data_input", color = "0.31 0.6 0.85", style="rounded"];
40[label = "build_industry_demand", color = "0.62 0.6 0.85", style="rounded"];
41[label = "build_industrial_distribution_key", color = "0.19 0.6 0.85", style="rounded"];
42[label = "build_industrial_database", color = "0.01 0.6 0.85", style="rounded,dashed"];
43[label = "build_base_industry_totals\ndemand: AB\nplanning_horizons: 2030", color = "0.16 0.6 0.85", style="rounded"];
44[label = "prepare_airports", color = "0.33 0.6 0.85", style="rounded"];
45[label = "prepare_gas_network", color = "0.00 0.6 0.85", style="rounded"];
46[label = "copy_config", color = "0.48 0.6 0.85", style="rounded"];
1 -> 0
2 -> 1
4 -> 1
46 -> 1
3 -> 2
4 -> 2
5 -> 2
6 -> 2
10 -> 2
7 -> 6
4 -> 6
30 -> 6
38 -> 6
31 -> 6
40 -> 6
28 -> 6
44 -> 6
3 -> 6
10 -> 6
45 -> 6
8 -> 7
28 -> 7
9 -> 8
4 -> 8
10 -> 9
4 -> 9
11 -> 10
16 -> 10
4 -> 10
12 -> 11
4 -> 11
22 -> 11
16 -> 11
13 -> 12
23 -> 12
24 -> 12
25 -> 12
26 -> 12
18 -> 12
4 -> 12
17 -> 12
16 -> 12
27 -> 12
14 -> 13
15 -> 13
16 -> 13
17 -> 13
22 -> 13
15 -> 14
15 -> 16
18 -> 17
20 -> 17
16 -> 17
19 -> 18
16 -> 18
20 -> 19
16 -> 19
21 -> 20
16 -> 20
16 -> 22
18 -> 22
14 -> 23
15 -> 23
16 -> 23
17 -> 23
22 -> 23
14 -> 24
15 -> 24
16 -> 24
17 -> 24
22 -> 24
14 -> 25
15 -> 25
16 -> 25
17 -> 25
22 -> 25
14 -> 26
15 -> 26
16 -> 26
17 -> 26
22 -> 26
18 -> 27
22 -> 27
15 -> 27
16 -> 27
29 -> 28
10 -> 30
28 -> 30
31 -> 30
34 -> 30
35 -> 30
36 -> 30
37 -> 30
32 -> 31
10 -> 31
15 -> 31
16 -> 32
33 -> 32
15 -> 32
32 -> 34
10 -> 34
15 -> 34
34 -> 35
32 -> 36
10 -> 36
15 -> 36
32 -> 37
10 -> 37
15 -> 37
10 -> 38
28 -> 38
39 -> 38
31 -> 38
34 -> 38
41 -> 40
43 -> 40
42 -> 40
4 -> 40
10 -> 41
31 -> 41
42 -> 41
29 -> 43
10 -> 45
}](_images/graphviz-1372c2bb60ec851ee55dcd69af7a8155ee4e1dff.png)
In the terminal, this will show up as a list of jobs to be run:
Building DAG of jobs...
Job stats:
job count
---------------------------------- -------
add_electricity 1
add_export 1
add_extra_components 1
base_network 1
build_base_energy_totals 1
build_base_industry_totals 1
build_bus_regions 1
build_clustered_population_layouts 1
build_cop_profiles 1
build_demand_profiles 1
build_heat_demand 1
build_industrial_distribution_key 1
build_industry_demand 1
build_natura_raster 1
build_osm_network 1
build_population_layouts 1
build_powerplants 1
build_renewable_profiles 5
build_shapes 1
build_ship_profile 1
build_solar_thermal_profiles 1
build_temperature_profiles 1
clean_osm_data 1
cluster_network 1
copy_config 1
download_osm_data 1
override_respot 1
prepare_airports 1
prepare_energy_totals 1
prepare_gas_network 1
prepare_heat_data 1
prepare_network 1
prepare_ports 1
prepare_sector_network 1
prepare_transport_data 1
prepare_transport_data_input 1
prepare_urban_percent 1
retrieve_cost_data 1
retrieve_databundle_light 1
simplify_network 1
solve_sector_network 1
solve_sector_networks 1
total 46
Myopic Foresight Scenarios#
Configuration#
The configuration to run the tutorial for the myopic foresight scenario is present
in the test/config.test_myopic.yaml file.
foresight: myopic
Note
It is important to set the following flags retrieve_databundle and retrieve_databundle_sector
to false after the first run to prevent unnecessary re-downloads, as the files only need to be downloaded once.
enable:
retrieve_databundle: true
retrieve_databundle_sector: true
Scenarios can be defined like for electricity-only studies, but with additional
wildcard options. For the myopic foresight mode, the {planning_horizons} wildcard
defines the sequence of investment horizons.
Note
The myopic optimisation is only possible on the sector-coupled model
scenario:
simpl: [""]
clusters: [4]
planning_horizons: [2030] # investment years for myopic and perfect; or costs year for overnight
ll: ["c1"]
opts: ["Co2L-24H"]
sopts: ["144h"]
demand: ["DF"]
For allowed wildcard values, refer to Wildcards. Documentation for all options will be added successively to Configuration.
Execution#
To run the tutorial for the sector-coupled model with myopic foresight, you need to activate the pypsa-earth environment. You need to have installed PyPSA-Earth using the instructions provided in the Installation section. Make sure to be in the PyPSA-Earth root directory and run the following command
Note
It is good practice to perform a dry-run using the option -n, before you commit to a run:
.../pypsa-earth (pypsa-earth) $ snakemake solve_sector_networks -j2 --configfile test/config.myopic.yaml -n
.../pypsa-earth (pypsa-earth) $ conda activate pypsa-earth
.../pypsa-earth (pypsa-earth) $ snakemake solve_sector_networks -j2 --configfile test/config.myopic.yaml
which will result in additional jobs snakemake wants to run, which translates to the following workflow diagram which nicely outlines how the sequential pathway optimisation with myopic foresight is implemented in the workflow:
![digraph snakemake_dag {
graph[bgcolor=white, margin=0];
node[shape=box, style=rounded, fontname=sans, fontsize=10, penwidth=2];
edge[penwidth=2, color=grey];
0[label = "solve_all_networks_myopic", color = "0.24 0.6 0.85", style="rounded"];
1[label = "solve_network_myopic", color = "0.04 0.6 0.85", style="rounded"];
2[label = "add_existing_baseyear", color = "0.57 0.6 0.85", style="rounded"];
3[label = "add_export", color = "0.43 0.6 0.85", style="rounded"];
4[label = "prepare_ports", color = "0.22 0.6 0.85", style="rounded"];
5[label = "retrieve_cost_data\nyear: 2030", color = "0.59 0.6 0.85", style="rounded"];
6[label = "build_ship_profile\nh2export: 120", color = "0.14 0.6 0.85", style="rounded"];
7[label = "prepare_sector_network", color = "0.19 0.6 0.85", style="rounded"];
8[label = "override_respot\ndiscountrate: 0.071\nsopts: 24H", color = "0.34 0.6 0.85", style="rounded"];
9[label = "prepare_network\nll: c1\nopts: Co2L", color = "0.63 0.6 0.85", style="rounded"];
10[label = "add_extra_components", color = "0.55 0.6 0.85", style="rounded"];
11[label = "cluster_network\nclusters: 4", color = "0.36 0.6 0.85", style="rounded"];
12[label = "simplify_network\nsimpl: ", color = "0.28 0.6 0.85", style="rounded"];
13[label = "add_electricity", color = "0.23 0.6 0.85", style="rounded"];
14[label = "build_renewable_profiles\ntechnology: onwind", color = "0.58 0.6 0.85", style="rounded"];
15[label = "build_natura_raster", color = "0.62 0.6 0.85", style="rounded"];
16[label = "retrieve_databundle_light", color = "0.10 0.6 0.85", style="rounded"];
17[label = "build_shapes", color = "0.37 0.6 0.85", style="rounded"];
18[label = "build_powerplants", color = "0.06 0.6 0.85", style="rounded"];
19[label = "base_network", color = "0.30 0.6 0.85", style="rounded"];
20[label = "build_osm_network", color = "0.25 0.6 0.85", style="rounded"];
21[label = "clean_osm_data", color = "0.31 0.6 0.85", style="rounded"];
22[label = "download_osm_data", color = "0.12 0.6 0.85", style="rounded"];
23[label = "build_bus_regions", color = "0.18 0.6 0.85", style="rounded"];
24[label = "build_renewable_profiles\ntechnology: offwind-ac", color = "0.58 0.6 0.85", style="rounded"];
25[label = "build_renewable_profiles\ntechnology: offwind-dc", color = "0.58 0.6 0.85", style="rounded"];
26[label = "build_renewable_profiles\ntechnology: solar", color = "0.58 0.6 0.85", style="rounded"];
27[label = "build_renewable_profiles\ntechnology: hydro", color = "0.58 0.6 0.85", style="rounded"];
28[label = "build_demand_profiles", color = "0.65 0.6 0.85", style="rounded"];
29[label = "prepare_energy_totals\ndemand: DF\nplanning_horizons: 2030", color = "0.35 0.6 0.85", style="rounded"];
30[label = "build_base_energy_totals", color = "0.51 0.6 0.85", style="rounded"];
31[label = "prepare_heat_data", color = "0.48 0.6 0.85", style="rounded"];
32[label = "build_clustered_population_layouts", color = "0.40 0.6 0.85", style="rounded"];
33[label = "build_population_layouts\nplanning_horizons: 2030", color = "0.60 0.6 0.85", style="rounded"];
34[label = "prepare_urban_percent", color = "0.29 0.6 0.85", style="rounded"];
35[label = "build_temperature_profiles", color = "0.52 0.6 0.85", style="rounded"];
36[label = "build_cop_profiles", color = "0.01 0.6 0.85", style="rounded"];
37[label = "build_solar_thermal_profiles", color = "0.27 0.6 0.85", style="rounded"];
38[label = "build_heat_demand", color = "0.07 0.6 0.85", style="rounded"];
39[label = "prepare_transport_data", color = "0.61 0.6 0.85", style="rounded"];
40[label = "prepare_transport_data_input", color = "0.13 0.6 0.85", style="rounded"];
41[label = "build_industry_demand", color = "0.20 0.6 0.85", style="rounded"];
42[label = "build_industrial_distribution_key", color = "0.00 0.6 0.85", style="rounded"];
43[label = "build_industrial_database", color = "0.41 0.6 0.85", style="rounded"];
44[label = "build_base_industry_totals\ndemand: DF\nplanning_horizons: 2030", color = "0.45 0.6 0.85", style="rounded"];
45[label = "prepare_airports", color = "0.26 0.6 0.85", style="rounded"];
46[label = "prepare_gas_network", color = "0.66 0.6 0.85", style="rounded"];
47[label = "build_existing_heating_distribution", color = "0.21 0.6 0.85", style="rounded"];
48[label = "copy_config", color = "0.56 0.6 0.85", style="rounded"];
1 -> 0
2 -> 1
5 -> 1
48 -> 1
3 -> 2
18 -> 2
12 -> 2
11 -> 2
32 -> 2
5 -> 2
36 -> 2
47 -> 2
4 -> 3
5 -> 3
6 -> 3
7 -> 3
11 -> 3
8 -> 7
5 -> 7
31 -> 7
39 -> 7
32 -> 7
41 -> 7
29 -> 7
45 -> 7
4 -> 7
11 -> 7
46 -> 7
9 -> 8
29 -> 8
10 -> 9
5 -> 9
11 -> 10
5 -> 10
12 -> 11
17 -> 11
5 -> 11
13 -> 12
5 -> 12
23 -> 12
17 -> 12
14 -> 13
24 -> 13
25 -> 13
26 -> 13
27 -> 13
19 -> 13
5 -> 13
18 -> 13
17 -> 13
28 -> 13
15 -> 14
16 -> 14
17 -> 14
18 -> 14
23 -> 14
16 -> 15
16 -> 17
19 -> 18
21 -> 18
17 -> 18
20 -> 19
17 -> 19
21 -> 20
17 -> 20
22 -> 21
17 -> 21
17 -> 23
19 -> 23
15 -> 24
16 -> 24
17 -> 24
18 -> 24
23 -> 24
15 -> 25
16 -> 25
17 -> 25
18 -> 25
23 -> 25
15 -> 26
16 -> 26
17 -> 26
18 -> 26
23 -> 26
15 -> 27
16 -> 27
17 -> 27
18 -> 27
23 -> 27
19 -> 28
23 -> 28
16 -> 28
17 -> 28
30 -> 29
11 -> 31
29 -> 31
32 -> 31
35 -> 31
36 -> 31
37 -> 31
38 -> 31
33 -> 32
11 -> 32
16 -> 32
17 -> 33
34 -> 33
16 -> 33
33 -> 35
11 -> 35
16 -> 35
35 -> 36
33 -> 37
11 -> 37
16 -> 37
33 -> 38
11 -> 38
16 -> 38
11 -> 39
29 -> 39
40 -> 39
32 -> 39
35 -> 39
42 -> 41
44 -> 41
43 -> 41
5 -> 41
11 -> 42
32 -> 42
43 -> 42
30 -> 44
11 -> 46
32 -> 47
31 -> 47
}](_images/graphviz-c407c118807508005406fa8639637f881af36b35.png)
In the terminal, this will show up as a list of jobs to be run:
Building DAG of jobs...
Job stats:
job count
----------------------------------- -------
add_electricity 1
add_existing_baseyear 1
add_export 1
add_extra_components 1
base_network 1
build_base_energy_totals 1
build_base_industry_totals 1
build_bus_regions 1
build_clustered_population_layouts 1
build_cop_profiles 1
build_demand_profiles 1
build_existing_heating_distribution 1
build_heat_demand 1
build_industrial_database 1
build_industrial_distribution_key 1
build_industry_demand 1
build_natura_raster 1
build_osm_network 1
build_population_layouts 1
build_powerplants 1
build_renewable_profiles 5
build_shapes 1
build_ship_profile 1
build_solar_thermal_profiles 1
build_temperature_profiles 1
clean_osm_data 1
cluster_network 1
copy_config 1
download_osm_data 1
override_respot 1
prepare_airports 1
prepare_energy_totals 1
prepare_gas_network 1
prepare_heat_data 1
prepare_network 1
prepare_ports 1
prepare_sector_network 1
prepare_transport_data 1
prepare_transport_data_input 1
prepare_urban_percent 1
retrieve_cost_data 1
retrieve_databundle_light 1
simplify_network 1
solve_all_networks_myopic 1
solve_network_myopic 1
total 49
Scaling-Up#
If you now feel confident and want to tackle runs with larger temporal, technological and spatial scopes, you can adjust the configuration file to your needs. You can also check the Model customization for more information on how to customize the model.