from d2l import torch as d2l
import logging
# Use INFO level so the periodic Syne Tune tuning-status table appears,
# but use a clean format that drops the "INFO:syne_tune.tuner:" prefix.
logging.basicConfig(level=logging.INFO, format="%(message)s", force=True)
# Silence Syne Tune's import-time chatter about optional AWS dependencies
# (sagemaker, s3fs) and Ray Tune. We use the local PythonBackend, so those
# are not needed. Suppress both print() and logging.info() during imports.
import contextlib, io
_root = logging.getLogger()
_prev_level = _root.level
_root.setLevel(logging.WARNING)
try:
with contextlib.redirect_stdout(io.StringIO()):
from syne_tune.config_space import loguniform, randint
from syne_tune.backend.python_backend.python_backend import PythonBackend
from syne_tune.optimizer.baselines import RandomSearch
from syne_tune import Tuner, StoppingCriterion
from syne_tune.experiments import load_experiment
finally:
_root.setLevel(_prev_level)
# Silence the per-trial subprocess-command spam from local_backend and
# drop the per-trial scheduling / completion lines from the tuner logger.
# Keep the periodic "tuning status (last metric is reported)" updates so
# the reader can still see progress over time.
class _DropPerTrialNoise(logging.Filter):
_DROP = (
"results of trials will be saved",
"scheduled ",
"Trial trial_id ",
)
def filter(self, record):
msg = record.getMessage()
return not any(s in msg for s in self._DROP)
logging.getLogger("syne_tune.backend.local_backend").setLevel(logging.WARNING)
logging.getLogger("syne_tune.tuner").addFilter(_DropPerTrialNoise())21.3 Asynchronous Random Search
As we have seen in the previous Section 21.2, we might have to wait hours or even days before random search returns a good hyperparameter configuration, because of the expensive evaluation of hyperparameter configurations. In practice, we have often access to a pool of resources such as multiple GPUs on the same machine or multiple machines with a single GPU. This begs the question: How do we efficiently distribute random search?
In general, we distinguish between synchronous and asynchronous parallel hyperparameter optimization (see Figure 21.3.1). In the synchronous setting, we wait for all concurrently running trials to finish, before we start the next batch. Consider configuration spaces that contain hyperparameters such as the number of filters or number of layers of a deep neural network. Hyperparameter configurations that contain a larger number of layers of filters will naturally take more time to finish, and all other trials in the same batch will have to wait at synchronization points (grey area in Figure 21.3.1) before we can continue the optimization process.
In the asynchronous setting we immediately schedule a new trial as soon as resources become available. This will optimally exploit our resources, since we can avoid any synchronization overhead. For random search, each new hyperparameter configuration is chosen independently of all others, and in particular without exploiting observations from any prior evaluation. This means we can trivially parallelize random search asynchronously. This is not straightforward with more sophisticated methods that make decision based on previous observations (see Section 21.5). While we need access to more resources than in the sequential setting, asynchronous random search exhibits a linear speed-up, in that a certain performance is reached \(K\) times faster if \(K\) trials can be run in parallel.
In this section, we will look at asynchronous random search where trials are executed in multiple python processes on the same machine. Distributed job scheduling and execution is difficult to implement from scratch. We will use Syne Tune (Salinas et al. 2022), which provides us with a simple interface for asynchronous HPO. Syne Tune is designed to be run with different execution back-ends, and the interested reader is invited to study its simple APIs in order to learn more about distributed HPO.
21.3.1 Objective Function
First, we have to define a new objective function such that it now returns the performance back to Syne Tune via the report callback.
def hpo_objective_lenet_synetune(learning_rate, batch_size, max_epochs):
from d2l import torch as d2l
from syne_tune import Reporter
model = d2l.LeNet(lr=learning_rate, num_classes=10)
trainer = d2l.HPOTrainer(max_epochs=1, num_gpus=1)
data = d2l.FashionMNIST(batch_size=batch_size)
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
report = Reporter()
for epoch in range(1, max_epochs + 1):
if epoch == 1:
# Initialize the state of Trainer
trainer.fit(model=model, data=data)
else:
trainer.fit_epoch()
validation_error = d2l.numpy(trainer.validation_error().cpu())
report(epoch=epoch, validation_error=float(validation_error))Note that the PythonBackend of Syne Tune requires dependencies to be imported inside the function definition.
21.3.2 Asynchronous Scheduler
First, we define the number of workers that evaluate trials concurrently. We also need to specify how long we want to run random search, by defining an upper limit on the total wall-clock time.
# Each LeNet trial fits in well under 7 GB of GPU memory, so we can pack
# multiple trials per device. `PythonBackend(rotate_gpus=True)` (the
# default) round-robins trials across detected GPUs and falls back to
# sharing when `n_workers > num_gpus`. Allocate 7 GB per slot — this
# yields 3 slots on a 24 GB card and 4 slots on a 32 GB card after
# driver overhead, e.g. 4×24 GB → 12 slots; 2×32 GB → 8.
import torch
_GB = 1024 ** 3
n_workers = sum(
torch.cuda.get_device_properties(i).total_memory // (7 * _GB)
for i in range(torch.cuda.device_count())
) or 1
max_wallclock_time = 15 * 60 # 15 minutesNext, we state which metric we want to optimize and whether we want to minimize or maximize this metric. Namely, metric needs to correspond to the argument name passed to the report callback.
mode = "min"
metric = "validation_error"We use the configuration space from our previous example. In Syne Tune, this dictionary can also be used to pass constant attributes to the training script. We make use of this feature in order to pass max_epochs. Moreover, we specify the first configuration to be evaluated in initial_config.
config_space = {
"learning_rate": loguniform(1e-2, 1),
"batch_size": randint(32, 256),
"max_epochs": 10,
}
initial_config = {
"learning_rate": 0.1,
"batch_size": 128,
}Next, we need to specify the back-end for job executions. Here we just consider the distribution on a local machine where parallel jobs are executed as sub-processes. However, for large scale HPO, we could run this also on a cluster or cloud environment, where each trial consumes a full instance.
trial_backend = PythonBackend(
tune_function=hpo_objective_lenet_synetune,
config_space=config_space,
)We can now create the scheduler for asynchronous random search, which is similar in behaviour to our BasicScheduler from Section 21.2.
scheduler = RandomSearch(
config_space,
metric=metric,
mode=mode,
points_to_evaluate=[initial_config],
)max_resource_level = 10, as inferred from config_spaceMaster random_seed = 1931818659Syne Tune also features a Tuner, where the main experiment loop and bookkeeping is centralized, and interactions between scheduler and back-end are mediated.
stop_criterion = StoppingCriterion(max_wallclock_time=max_wallclock_time)
tuner = Tuner(
trial_backend=trial_backend,
scheduler=scheduler,
stop_criterion=stop_criterion,
n_workers=n_workers,
print_update_interval=int(max_wallclock_time * 0.6),
)Let us run our distributed HPO experiment. According to our stopping criterion, it will run for about 15 minutes.
tuner.run()tuning status (last metric is reported)
trial_id status iter learning_rate batch_size max_epochs epoch validation_error worker-time
0 Completed 10 0.100000 128 10 10 0.265922 53.745338
1 Completed 10 0.144649 162 10 10 0.306047 43.769002
2 Completed 10 0.024609 240 10 10 0.899950 38.470056
3 Completed 10 0.326677 136 10 10 0.234897 42.585098
4 Completed 10 0.053636 136 10 10 0.371666 49.575488
5 Completed 10 0.013939 118 10 10 0.900027 59.184844
6 Completed 10 0.121661 93 10 10 0.250098 60.566196
7 Completed 10 0.479110 158 10 10 0.178677 42.156294
8 Completed 10 0.205486 63 10 10 0.195511 74.810880
9 Completed 10 0.048735 176 10 10 0.483697 44.501192
10 Completed 10 0.558970 143 10 10 0.193003 46.308470
11 Completed 10 0.014408 178 10 10 0.900989 42.185262
12 Completed 10 0.019423 47 10 10 0.369507 88.139082
13 Completed 10 0.242102 81 10 10 0.188205 64.994654
14 Completed 10 0.066219 71 10 10 0.296268 51.761426
15 Completed 10 0.281574 243 10 10 0.253825 38.683142
16 Completed 10 0.212848 150 10 10 0.226667 42.883267
17 Completed 10 0.012426 197 10 10 0.884983 41.411002
18 Completed 10 0.213072 180 10 10 0.237579 41.413629
19 Completed 10 0.012028 219 10 10 0.900076 40.047592
20 Completed 10 0.170024 93 10 10 0.262921 49.081692
21 Completed 10 0.380933 202 10 10 0.388154 40.089510
22 Completed 10 0.212603 201 10 10 0.261550 39.770660
23 Completed 10 0.102020 222 10 10 0.399785 40.179164
24 Completed 10 0.882282 190 10 10 0.190980 40.708301
25 InProgress 7 0.051366 70 10 7 0.347736 61.546086
26 InProgress 10 0.027160 241 10 10 0.900091 38.281000
27 InProgress 4 0.024477 53 10 4 0.899887 27.743438
3 trials running, 25 finished (25 until the end), 541.45s wallclock-timereaching max wallclock time (900), stopping there.--------------------
Resource summary (last result is reported):
trial_id status iter learning_rate batch_size max_epochs epoch validation_error worker-time
0 Completed 10 0.100000 128 10 10 0.265922 53.745338
1 Completed 10 0.144649 162 10 10 0.306047 43.769002
2 Completed 10 0.024609 240 10 10 0.899950 38.470056
3 Completed 10 0.326677 136 10 10 0.234897 42.585098
4 Completed 10 0.053636 136 10 10 0.371666 49.575488
5 Completed 10 0.013939 118 10 10 0.900027 59.184844
6 Completed 10 0.121661 93 10 10 0.250098 60.566196
7 Completed 10 0.479110 158 10 10 0.178677 42.156294
8 Completed 10 0.205486 63 10 10 0.195511 74.810880
9 Completed 10 0.048735 176 10 10 0.483697 44.501192
10 Completed 10 0.558970 143 10 10 0.193003 46.308470
11 Completed 10 0.014408 178 10 10 0.900989 42.185262
12 Completed 10 0.019423 47 10 10 0.369507 88.139082
13 Completed 10 0.242102 81 10 10 0.188205 64.994654
14 Completed 10 0.066219 71 10 10 0.296268 51.761426
15 Completed 10 0.281574 243 10 10 0.253825 38.683142
16 Completed 10 0.212848 150 10 10 0.226667 42.883267
17 Completed 10 0.012426 197 10 10 0.884983 41.411002
18 Completed 10 0.213072 180 10 10 0.237579 41.413629
19 Completed 10 0.012028 219 10 10 0.900076 40.047592
20 Completed 10 0.170024 93 10 10 0.262921 49.081692
21 Completed 10 0.380933 202 10 10 0.388154 40.089510
22 Completed 10 0.212603 201 10 10 0.261550 39.770660
23 Completed 10 0.102020 222 10 10 0.399785 40.179164
24 Completed 10 0.882282 190 10 10 0.190980 40.708301
25 Completed 10 0.051366 70 10 10 0.287246 85.876910
26 Completed 10 0.027160 241 10 10 0.900091 38.281000
27 Completed 10 0.024477 53 10 10 0.381770 66.921683
28 Completed 10 0.507380 188 10 10 0.268847 36.950613
29 Completed 10 0.016139 167 10 10 0.900037 42.718961
30 Completed 10 0.317496 157 10 10 0.267815 38.672534
31 Completed 10 0.087387 115 10 10 0.275980 42.198973
32 Completed 10 0.060654 157 10 10 0.413553 41.080449
33 Completed 10 0.153702 122 10 10 0.235394 52.161024
34 Completed 10 0.403919 55 10 10 0.153746 67.003750
35 Completed 10 0.012223 164 10 10 0.900000 39.698314
36 Completed 10 0.100816 142 10 10 0.274102 40.892768
37 Completed 10 0.071518 201 10 10 0.460780 38.902775
38 Completed 10 0.051194 103 10 10 0.353565 45.783863
39 Completed 10 0.133720 238 10 10 0.299052 37.986270
40 Completed 10 0.027542 129 10 10 0.759419 42.043354
41 Completed 10 0.637011 78 10 10 0.278747 51.347403
42 Completed 10 0.418927 172 10 10 0.203439 39.878935
43 Completed 10 0.839018 213 10 10 0.193084 40.106312
44 Completed 10 0.010611 129 10 10 0.900248 44.373506
45 Completed 10 0.260142 169 10 10 0.413507 42.530311
46 InProgress 7 0.061649 191 10 7 0.900500 28.085097
47 InProgress 3 0.112213 211 10 3 0.899589 12.437097
48 InProgress 2 0.096105 145 10 2 0.900021 9.008643
3 trials running, 46 finished (46 until the end), 903.41s wallclock-time
validation_error: best 0.14445585012435913 for trial-id 34
--------------------Stopping trials that may still be running.Tuning finished, results of trials can be found on /home/smola/syne-tune/python-entrypoint-2026-07-19-13-45-15-096The logs of all evaluated hyperparameter configurations are stored for further analysis. At any time during the tuning job, we can easily get the results obtained so far and plot the incumbent trajectory.
d2l.set_figsize()
tuning_experiment = load_experiment(tuner.name)
tuning_experiment.plot()
21.3.3 Visualize the Asynchronous Optimization Process
Below we visualize how the learning curves of every trial (each color in the plot represents a trial) evolve during the asynchronous optimization process. At any point in time, there are as many trials running concurrently as we have workers. Once a trial finishes, we immediately start the next trial, without waiting for the other trials to finish. Idle time of workers is reduced to a minimum with asynchronous scheduling.
d2l.set_figsize([6, 2.5])
results = tuning_experiment.results
for trial_id in results.trial_id.unique():
df = results[results["trial_id"] == trial_id]
d2l.plt.plot(
df["st_tuner_time"],
df["validation_error"],
marker="o"
)
d2l.plt.xlabel("wall-clock time")
d2l.plt.ylabel("objective function")Text(0, 0.5, 'objective function')
21.3.4 Summary
We can reduce the waiting time for random search substantially by distributing trials across parallel resources. In general, we distinguish between synchronous scheduling and asynchronous scheduling. Synchronous scheduling means that we sample a new batch of hyperparameter configurations once the previous batch finished. If we have stragglers - trials that take more time to finish than other trials - our workers need to wait at synchronization points. Asynchronous scheduling evaluates new hyperparameter configurations as soon as resources become available, and, hence, ensures that all workers are busy at any point in time. While random search is easy to distribute asynchronously and does not require any change of the actual algorithm, other methods require some additional modifications.
21.3.5 Exercises
- Consider the
DropoutMLPmodel implemented in Section 4.6, and used in Exercise 1 of Section 21.2.- Implement an objective function
hpo_objective_dropoutmlp_synetuneto be used with Syne Tune. Make sure that your function reports the validation error after every epoch. - Using the setup of Exercise 1 in Section 21.2, compare random search to Bayesian optimization. If you use SageMaker, feel free to use Syne Tune’s benchmarking facilities in order to run experiments in parallel. Hint: Bayesian optimization is provided as
syne_tune.optimizer.baselines.BayesianOptimization. - For this exercise, you need to run on an instance with at least 4 CPU cores. For one of the methods used above (random search, Bayesian optimization), run experiments with
n_workers=1,n_workers=2,n_workers=4, and compare results (incumbent trajectories). At least for random search, you should observe linear scaling with respect to the number of workers. Hint: For robust results, you may have to average over several repetitions each.
- Implement an objective function
- Advanced. The goal of this exercise is to implement a new scheduler in Syne Tune.
- Create a virtual environment containing both the d2lbook and syne-tune sources.
- Implement the
LocalSearcherfrom Exercise 2 in Section 21.2 as a new searcher in Syne Tune. Hint: Read this tutorial. Alternatively, you may follow this example. - Compare your new
LocalSearcherwithRandomSearchon theDropoutMLPbenchmark.