4. Manager Configuration Files¶
This page contains a centralized reference for all of the configuration options
in config_runtime.ini
, config_build.ini
, config_build_recipes.ini
,
and config_hwdb.ini
.
4.1. config_runtime.ini
¶
Here is a sample of this configuration file:
# RUNTIME configuration for the FireSim Simulation Manager
# See docs/Advanced-Usage/Manager/Manager-Configuration-Files.rst for
# documentation of all of these params.
[runfarm]
runfarmtag=mainrunfarm
f1_16xlarges=1
m4_16xlarges=0
f1_4xlarges=0
f1_2xlarges=0
runinstancemarket=ondemand
spotinterruptionbehavior=terminate
spotmaxprice=ondemand
[targetconfig]
#Set topology=no_net_config to run without a network simulation
topology=example_8config
no_net_num_nodes=2
linklatency=6405
switchinglatency=10
netbandwidth=200
profileinterval=-1
# This references a section from config_hwconfigs.ini
# In homogeneous configurations, use this to set the hardware config deployed
# for all simulators
defaulthwconfig=firesim-rocket-quadcore-nic-l2-llc4mb-ddr3
[tracing]
enable=no
# Trace output formats. Only enabled if "enable" is set to "yes" above
# 0 = human readable; 1 = binary (compressed raw data); 2 = flamegraph (stack
# unwinding -> Flame Graph)
output_format=0
# Trigger selector.
# 0 = no trigger; 1 = cycle count trigger; 2 = program counter trigger; 3 =
# instruction trigger
selector=1
start=0
end=-1
[autocounter]
readrate=0
[workload]
workloadname=linux-uniform.json
terminateoncompletion=no
suffixtag=
[hostdebug]
# When enabled (=yes), Zeros-out FPGA-attached DRAM before simulations
# begin (takes 2-5 minutes).
# In general, this is not required to produce deterministic simulations on
# target machines running linux. Enable if you observe simulation non-determinism.
zerooutdram=no
[synthprint]
# Start and end cycles for outputting synthesized prints.
# They are given in terms of the base clock and will be converted
# for each clock domain.
start=0
end=-1
# When enabled (=yes), prefix print output with the target cycle at which the print was triggered
cycleprefix=yes
Below, we outline each section and parameter in detail.
4.1.1. [runfarm]
¶
The [runfarm]
options below allow you to specify the number, types, and
other characteristics of instances in your FireSim Run Farm, so that the
manager can automatically launch them, run workloads on them, and terminate
them.
4.1.1.1. runfarmtag
¶
Use runfarmtag
to differentiate between different Run Farms in FireSim.
Having multiple config_runtime.ini
files with different runfarmtag
values allows you to run many experiments at once from the same manager instance.
The instances launched by the launchrunfarm
command will be tagged with
this value. All later operations done by the manager rely on this tag, so
you should not change it unless you are done with your current Run Farm.
Per AWS restrictions, this tag can be no longer than 255 characters.
4.1.1.2. f1_16xlarges
, m4_16xlarges
, f1_4xlarges
, f1_2xlarges
¶
Set these three values respectively based on the number and types of instances you need. While we could automate this setting, we choose not to, so that users are never surprised by how many instances they are running.
Note that these values are ONLY used to launch instances. After launch, the
manager will query the AWS API to find the instances of each type that have the
runfarmtag
set above assigned to them.
4.1.1.3. runinstancemarket
¶
You can specify either spot
or ondemand
here, to use one of those
markets on AWS.
4.1.1.4. spotinterruptionbehavior
¶
When runinstancemarket=spot
, this value determines what happens to an instance
if it receives the interruption signal from AWS. You can specify either
hibernate
, stop
, or terminate
.
4.1.1.5. spotmaxprice
¶
When runinstancemarket=spot
, this value determines the max price you are
willing to pay per instance, in dollars. You can also set it to ondemand
to set your max to the on-demand price for the instance.
4.1.2. [targetconfig]
¶
The [targetconfig]
options below allow you to specify the high-level
configuration of the target you are simulating. You can change these parameters
after launching a Run Farm (assuming you have the correct number of instances),
but in many cases you will need to re-run the infrasetup
command to make
sure the correct simulation infrastructure is available on your instances.
4.1.2.1. topology
¶
This field dictates the network topology of the simulated system. Some examples:
no_net_config
: This runs N (see no_net_num_nodes
below) independent
simulations, without a network simulation. You can currently only use this
option if you build one of the NoNIC hardware configs of FireSim.
example_8config
: This requires a single f1.16xlarge
, which will
simulate 1 ToR switch attached to 8 simulated servers.
example_16config
: This requires two f1.16xlarge
instances and one
m4.16xlarge
instance, which will
simulate 2 ToR switches, each attached to 8 simulated servers, with the two
ToR switches connected by a root switch.
example_64config
: This requires eight f1.16xlarge
instances and one
m4.16xlarge
instance, which will simulate 8 ToR switches, each attached to
8 simulated servers (for a total of 64 nodes), with the eight ToR switches
connected by a root switch.
Additional configurations are available in deploy/runtools/user_topology.py
and more can be added there. See the Manager Network Topology Definitions (user_topology.py) section
for more info.
4.1.2.2. no_net_num_nodes
¶
This determines the number of simulated nodes when you are using
topology=no_net_config
.
4.1.2.3. linklatency
¶
In a networked simulation, this allows you to specify the link latency of the simulated network in CYCLES. For example, 6405 cycles is roughly 2 microseconds at 3.2 GHz. A current limitation is that this value (in cycles) must be a multiple of 7. Furthermore, you must not exceed the buffer size specified in the NIC’s simulation widget.
4.1.2.4. switchinglatency
¶
In a networked simulation, this specifies the minimum port-to-port switching latency of the switch models, in CYCLES.
4.1.2.5. netbandwidth
¶
In a networked simulation, this specifies the maximum output bandwidth that a NIC is allowed to produce as an integer in Gbit/s. Currently, this must be a number between 1 and 200, allowing you to model NICs between 1 and 200 Gbit/s.
4.1.2.6. profileinterval
¶
The simulation driver periodically samples performance counters in FASED timing model instances and dumps the result to a file on the host. profileinterval
defines the number of target cycles between samples; setting this field to -1 disables polling.
4.1.2.7. defaulthwconfig
¶
This sets the server configuration launched by default in the above topologies.
Heterogeneous configurations can be achieved by manually specifying different
names within the topology itself, but all the example_Nconfig
configurations
are homogeneous and use this value for all nodes.
You should set this to one of the hardware configurations you have defined already in
config_hwdb.ini
. You should set this to the NAME (section title) of the
hardware configuration from config_hwdb.ini
, NOT the actual agfi itself
(NOT something like agfi-XYZ...
).
4.1.3. [tracing]
¶
This section manages TracerV-based tracing at simulation runtime. For more details, see the Capturing RISC-V Instruction Traces with TracerV page for more details.
4.1.3.1. enable
¶
This turns tracing on, when set to yes
and off when set to no
. See the Enabling Tracing at Runtime.
4.1.3.2. output_format
¶
This sets the output format for TracerV tracing. See the Selecting a Trace Output Format section.
4.1.3.3. selector
, start
, and end
¶
These configure triggering for TracerV. See the Setting a TracerV Trigger section.
4.1.4. [autocounter]
¶
This section configures AutoCounter. See the AutoCounter: Profiling with Out-of-Band Performance Counter Collection page for more details.
4.1.4.1. readrate
¶
This sets the rate at which AutoCounters are read. See the AutoCounter Runtime Parameters section for more details.
4.1.5. [workload]
¶
This section defines the software that will run on the simulated system.
4.1.5.1. workloadname
¶
This selects a workload to run across the set of simulated nodes. A workload consists of a series of jobs that need to be run on simulated nodes (one job per node).
Workload definitions are located in firesim/deploy/workloads/*.json
.
Some sample workloads:
linux-uniform.json
: This runs the default FireSim Linux distro on as many nodes
as you specify when setting the [targetconfig]
parameters.
spec17-intrate.json
: This runs SPECint 2017’s rate benchmarks. In this type of
workload, you should launch EXACTLY the correct number of nodes necessary to run the
benchmark. If you specify fewer nodes, the manager will warn that not all jobs were
assigned to a simulation. If you specify too many simulations and not enough
jobs, the manager will not launch the jobs.
Others can be found in the aforementioned directory. For a description of the JSON format, see Defining Custom Workloads.
4.1.5.2. terminateoncompletion
¶
Set this to no
if you want your Run Farm to keep running once the workload
has completed. Set this to yes
if you want your Run Farm to be TERMINATED
after the workload has completed and results have been copied off.
4.1.5.3. suffixtag
¶
This allows you to append a string to a workload’s output directory name,
useful for differentiating between successive runs of the same workload,
without renaming the entire workload. For example, specifying
suffixtag=test-v1
with a workload named super-application
will result
in a workload results directory named
results-workload/DATE--TIME-super-application-test-v1/
.
4.1.6. [hostdebug]
¶
4.1.6.1. zerooutdram
¶
Set this to yes
to zero-out FPGA-attached DRAM before simulation begins.
This process takes 2-5 minutes. In general, this is not required to produce
deterministic simulations on target machines running linux, but should be
enabled if you observe simulation non-determinism.
4.2. config_build.ini
¶
Here is a sample of this configuration file:
# BUILDTIME/AGFI management configuration for the FireSim Simulation Manager
# See docs/Advanced-Usage/Manager/Manager-Configuration-Files.rst for documentation of all of these params.
[afibuild]
s3bucketname=firesim-AWSUSERNAME
buildinstancemarket=ondemand
spotinterruptionbehavior=terminate
spotmaxprice=ondemand
postbuildhook=
[builds]
# this section references builds defined in config_build_recipes.ini
# if you add a build here, it will be built when you run buildafi
firesim-rocket-quadcore-nic-l2-llc4mb-ddr3
firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3
firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3
firesim-boom-singlecore-nic-l2-llc4mb-ddr3
firesim-supernode-rocket-singlecore-nic-l2-lbp
#firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3-ramopts
firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3-half-freq-uncore
# SHA3 configs for tutorial
# firesim-singlecore-sha3-no-nic-l2-llc4mb-ddr3
# firesim-singlecore-sha3-print-no-nic-l2-llc4mb-ddr3
[agfistoshare]
firesim-rocket-quadcore-nic-l2-llc4mb-ddr3
firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3
firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3
firesim-boom-singlecore-nic-l2-llc4mb-ddr3
firesim-supernode-rocket-singlecore-nic-l2-lbp
#firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3-ramopts
firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3-half-freq-uncore
# SHA3 configs for tutorial
# firesim-singlecore-sha3-no-nic-l2-llc4mb-ddr3
# firesim-singlecore-sha3-print-no-nic-l2-llc4mb-ddr3
[sharewithaccounts]
# To share with a specific user:
somebodysname=123456789012
# To share publicly:
#public=public
Below, we outline each section and parameter in detail.
4.2.1. [afibuild]
¶
This exposes options for AWS resources used in the process of building FireSim AGFIs (FPGA Images).
4.2.1.1. s3bucketname
¶
This is used behind the scenes in the AGFI creation process. You will only ever need to access this bucket manually if there is a failure in AGFI creation in Amazon’s backend.
Naming rules: this must be all lowercase and you should stick to letters and numbers.
The first time you try to run a build, the FireSim manager will try to create the bucket you name here. If the name is unavailable, it will complain and you will need to change this name. Once you choose a working name, you should never need to change it.
In general, firesim-yournamehere
is a good choice.
4.2.1.2. buildinstancemarket
¶
You can specify either spot
or ondemand
here, to use one of those
markets on AWS.
4.2.1.3. spotinterruptionbehavior
¶
When buildinstancemarket=spot
, this value determines what happens to an
instance if it receives the interruption signal from AWS. You can specify
either hibernate
, stop
, or terminate
.
4.2.1.4. spotmaxprice
¶
When buildinstancemarket=spot
, this value determines the max price you are
willing to pay per instance, in dollars. You can also set it to ondemand
to set your max to the on-demand price for the instance.
4.2.1.5. postbuildhook
¶
(Optional) Provide an a script to run on the results copied back from a _single_ build instance. Upon completion of each design’s build, the manager invokes this script and passing the absolute path to that instance’s build-results directory as it’s first argument.
4.2.2. [builds]
¶
In this section, you can list as many build entries as you want to run
for a particular call to the buildafi
command (see
config_build_recipes.ini
below for how to define a build entry). For
example, if we want to run the builds named [awesome-firesim-config]
and [quad-core-awesome-firesim-config]
, we would
write:
[builds]
awesome-firesim-config
quad-core-awesome-firesim-config
4.3. config_build_recipes.ini
¶
Here is a sample of this configuration file:
# Build-time design configuration for the FireSim Simulation Manager # See docs/Advanced-Usage/Manager/Manager-Configuration-Files.rst for documentation of all of these params. # this file contains sections that describe hardware designs that /can/ be built. # edit config_build.ini to actually "turn on" a config to be built when you run # buildafi # Note: For large designs (ones that would fill a EC2.2xlarge/Xilinx VU9P) # Vivado uses in excess of 32 GiB. Keep this in mind when selecting a # non-default instancetype. # Quad-core, Rocket-based recipes [firesim-rocket-quadcore-nic-l2-llc4mb-ddr3] DESIGN=FireSim TARGET_CONFIG=WithNIC_DDR3FRFCFSLLC4MB_FireSimQuadRocketConfig PLATFORM_CONFIG=F90MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None [firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3] DESIGN=FireSim TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimQuadRocketConfig PLATFORM_CONFIG=F90MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # Single-core, BOOM-based recipes [firesim-boom-singlecore-nic-l2-llc4mb-ddr3] DESIGN=FireSim TARGET_CONFIG=WithNIC_DDR3FRFCFSLLC4MB_FireSimLargeBoomConfig PLATFORM_CONFIG=F65MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None [firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3] DESIGN=FireSim TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimLargeBoomConfig PLATFORM_CONFIG=F70MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # Single-core, Ariane-based recipes [firesim-ariane-singlecore-no-nic-l2-llc4mb-ddr3] DESIGN=FireSim TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimArianeConfig PLATFORM_CONFIG=F90MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # Single-core, Rocket-based recipes with Gemmini [firesim-rocket-singlecore-gemmini-no-nic-l2-llc4mb-ddr3] DESIGN=FireSim TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimGemminiRocketConfig PLATFORM_CONFIG=F110MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # RAM Optimizations enabled by adding _MCRams PLATFORM_CONFIG string [firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3-ramopts] DESIGN=FireSim TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimLargeBoomConfig PLATFORM_CONFIG=MCRams_F90MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # Supernode configurations -- multiple instances of an SoC in a single simulator [firesim-supernode-rocket-singlecore-nic-l2-lbp] DESIGN=FireSim TARGET_CONFIG=WithNIC_SupernodeFireSimRocketConfig PLATFORM_CONFIG=F85MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # Multiclock Temporary Example:ncept Quad-core, Rocket-based recipes [firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3-half-freq-uncore] DESIGN=FireSimMulticlockPOC TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimQuadRocketMulticlockConfig PLATFORM_CONFIG=F90MHz_BaseF1Config instancetype=z1d.2xlarge deploytriplet=None # MIDAS Examples -- BUILD SUPPORT ONLY; Can't launch driver correctly on runfarm [midasexamples-gcd] TARGET_PROJECT=midasexamples DESIGN=GCD TARGET_CONFIG=NoConfig PLATFORM_CONFIG=DefaultF1Config instancetype=z1d.2xlarge deploytriplet=None
Below, we outline each section and parameter in detail.
4.3.1. Build definition sections, e.g. [awesome-firesim-config]
¶
In this file, you can specify as many build definition sections as you want,
each with a header like [awesome-firesim-config]
(i.e. a nice, short name
you made up). Such a section must contain the following fields:
4.3.1.1. DESIGN
¶
This specifies the basic target design that will be built. Unless you
are defining a custom system, this should be set to FireSim
.
We describe this in greater detail in Generating Different
Targets).
4.3.1.2. TARGET_CONFIG
¶
This specifies the hardware configuration of the target being simulated. Some
examples include FireSimRocketConfig
and FireSimQuadRocketConfig
.
We describe this in greater detail in Generating Different
Targets).
4.3.1.3. PLATFORM_CONFIG
¶
This specifies hardware parameters of the simulation environment - for example,
selecting between a Latency-Bandwidth Pipe or DDR3 memory models.
These are defined in firesim/sim/src/main/scala/firesim/SimConfigs.scala
.
We specify the host FPGA frequency in the PLATFORM_CONFIG
by appending a frequency
Config
with an underscore (ex. BaseF1Config_F160MHz).
We describe this in greater detail in Generating Different
Targets).
4.3.1.4. instancetype
¶
This defines the type of instance that the build will run on. Generally, running
on a z1d.2xlarge
is sufficient. In our experience, using more powerful instances
than this provides little gain.
4.3.1.5. deploytriplet
¶
This allows you to override the deploytriplet
stored with the AGFI.
Otherwise, the DESIGN
/TARGET_CONFIG
/PLATFORM_CONFIG
you specify
above will be used. See the AGFI Tagging section for more details. Most likely,
you should leave this set to None
. This is usually only used if you have
proprietary RTL that you bake into an FPGA image, but don’t want to share with
users of the simulator.
4.3.1.6. TARGET_PROJECT
(Optional)¶
This specifies the target project in which the target is defined (this is described
in greater detail here). If
TARGET_PROJECT
is undefined the manager will default to firesim
.
Setting TARGET_PROJECT
is required for building the MIDAS examples
(TARGET_PROJECT=midasexamples
) with the manager, or for building a
user-provided target project.
4.4. config_hwdb.ini
¶
Here is a sample of this configuration file:
# Hardware config database for FireSim Simulation Manager # See docs/Advanced-Usage/Manager/Manager-Configuration-Files.rst for documentation of all of these params. # Hardware configs represent a combination of an agfi, a deploytriplet override # (if needed), and a custom runtime config (if needed) # The AGFIs provided below are public and available to all users. # Only AGFIs for the latest release of FireSim are guaranteed to be available. # If you are using an older version of FireSim, you will need to generate your # own images. [firesim-boom-singlecore-nic-l2-llc4mb-ddr3] agfi=agfi-02c64d9441726e956 deploytripletoverride=None customruntimeconfig=None [firesim-boom-singlecore-no-nic-l2-llc4mb-ddr3] agfi=agfi-01057aa9ca7f38969 deploytripletoverride=None customruntimeconfig=None [firesim-rocket-quadcore-nic-l2-llc4mb-ddr3] agfi=agfi-009e7a5f7c315a6ec deploytripletoverride=None customruntimeconfig=None [firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3] agfi=agfi-0e6e54cd1923411cf deploytripletoverride=None customruntimeconfig=None [firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3-half-freq-uncore] agfi=agfi-0c52d2901b07eccad deploytripletoverride=None customruntimeconfig=None [firesim-supernode-rocket-singlecore-nic-l2-lbp] agfi=agfi-0d814ea4b81eb484e deploytripletoverride=None customruntimeconfig=None
This file tracks hardware configurations that you can deploy as simulated nodes in FireSim. Each such configuration contains a name for easy reference in higher-level configurations, defined in the section header, an agfi, which represents the FPGA image, a custom runtime config, if one is needed, and a deploy triplet override if one is necessary.
When you build a new AGFI, you should put the default version of it in this file so that it can be referenced from your other configuration files.
The following is an example section from this file - you can add as many of these as necessary:
[firesim-rocket-quadcore-nic-l2-llc4mb-ddr3]
# this is a comment that describes my favorite configuration!
agfi=agfi-0a6449b5894e96e53
deploytripletoverride=None
customruntimeconfig=None
4.4.1. [NAME_GOES_HERE]
¶
In this example, firesim-rocket-quadcore-nic-l2-llc4mb-ddr3
is the name that will be
used to reference this hardware design in other configuration locations. The following
items describe this hardware configuration:
4.4.1.1. agfi
¶
This represents the AGFI (FPGA Image) used by this hardware configuration.
4.4.1.2. deploytripletoverride
¶
This is an advanced feature - under normal conditions, you should leave this set to None
, so that the
manager uses the configuration triplet that is automatically stored with the
AGFI at build time. Advanced users can set this to a different
value to build and use a different driver when deploying simulations. Since
the driver depends on logic now hardwired into the
FPGA bitstream, drivers cannot generally be changed without requiring FPGA
recompilation.
4.4.1.3. customruntimeconfig
¶
This is an advanced feature - under normal conditions, you can use the default
parameters generated automatically by the simulator by setting this field to
None
. If you want to customize runtime parameters for certain parts of
the simulation (e.g. the DRAM model’s runtime parameters), you can place
a custom config file in sim/custom-runtime-configs/
. Then, set this field
to the relative name of the config. For example,
sim/custom-runtime-configs/GREATCONFIG.conf
becomes
customruntimeconfig=GREATCONFIG.conf
.
4.4.2. Add more hardware config sections, like [NAME_GOES_HERE_2]
¶
You can add as many of these entries to config_hwdb.ini
as you want, following the format
discussed above (i.e. you provide agfi
, deploytripletoverride
, or customruntimeconfig
).