from d2l import torch as d2l
import logging
# Use INFO level so the periodic Syne Tune tuning-status table appears,
# but use a clean format that drops the "INFO:syne_tune.tuner:" prefix.
logging.basicConfig(level=logging.INFO, format="%(message)s", force=True)
import matplotlib.pyplot as plt
# Silence Syne Tune's import-time chatter about optional AWS dependencies
# (sagemaker, s3fs) and Ray Tune. We use the local PythonBackend, so those
# are not needed. Suppress both print() and logging.info() during imports.
import contextlib, io
_root = logging.getLogger()
_prev_level = _root.level
_root.setLevel(logging.WARNING)
try:
with contextlib.redirect_stdout(io.StringIO()):
from syne_tune.config_space import loguniform, randint
from syne_tune.backend.python_backend.python_backend import PythonBackend
from syne_tune.optimizer.baselines import ASHA
from syne_tune import Tuner, StoppingCriterion
from syne_tune.experiments import load_experiment
finally:
_root.setLevel(_prev_level)
# Silence the per-trial subprocess-command spam from local_backend and
# drop the per-trial scheduling / completion lines from the tuner logger.
# Keep the periodic "tuning status (last metric is reported)" updates so
# the reader can still see progress over time.
class _DropPerTrialNoise(logging.Filter):
_DROP = (
"results of trials will be saved",
"scheduled ",
"Trial trial_id ",
)
def filter(self, record):
msg = record.getMessage()
return not any(s in msg for s in self._DROP)
logging.getLogger("syne_tune.backend.local_backend").setLevel(logging.WARNING)
logging.getLogger("syne_tune.tuner").addFilter(_DropPerTrialNoise())21.5 Asynchronous Successive Halving
As we have seen in Section 21.3, we can accelerate HPO by distributing the evaluation of hyperparameter configurations across either multiple instances or multiple CPUs / GPUs on a single instance. However, compared to random search, it is not straightforward to run successive halving (SH) asynchronously in a distributed setting. Before we can decide which configuration to run next, we first have to collect all observations at the current rung level. This requires to synchronize workers at each rung level. For example, for the lowest rung level \(r_{\mathrm{min}}\), we first have to evaluate all \(N = \eta^K\) configurations, before we can promote the \(\frac{1}{\eta}\) of them to the next rung level.
In any distributed system, synchronization typically implies idle time for workers. First, we often observe high variations in training time across hyperparameter configurations. For example, assuming the number of filters per layer is a hyperparameter, then networks with less filters finish training faster than networks with more filters, which implies idle worker time due to stragglers. Moreover, the number of slots in a rung level is not always a multiple of the number of workers, in which case some workers may even sit idle for a full batch.
Figure Figure 21.5.1 shows the scheduling of synchronous SH with \(\eta=2\) for four different trials with two workers. We start with evaluating Trial-0 and Trial-1 for one epoch and immediately continue with the next two trials once they are finished. We first have to wait until Trial-2 finishes, which takes substantially more time than the other trials, before we can promote the best two trials, i.e., Trial-0 and Trial-3 to the next rung level. This causes idle time for Worker-1. Then, we continue with Rung 1. Also, here Trial-3 takes longer than Trial-0, which leads to an additional idling time of Worker-0. Once we reach Rung-2, only the best trial, Trial-0, remains which occupies only one worker. To avoid that Worker-1 idles during that time, most implementations of SH continue already with the next round, and start evaluating new trials (e.g Trial-4) on the first rung.
Asynchronous successive halving (ASHA) (Li et al. 2018) adapts SH to the asynchronous parallel scenario. The main idea of ASHA is to promote configurations to the next rung level as soon as we collected at least \(\eta\) observations on the current rung level. This decision rule may lead to suboptimal promotions: configurations can be promoted to the next rung level, which in hindsight do not compare favourably against most others at the same rung level. On the other hand, we get rid of all synchronization points this way. In practice, such suboptimal initial promotions have only a modest impact on performance, not only because the ranking of hyperparameter configurations is often fairly consistent across rung levels, but also because rungs grow over time and reflect the distribution of metric values at this level better and better. If a worker is free, but no configuration can be promoted, we start a new configuration with \(r = r_{\mathrm{min}}\), i.e the first rung level.
Figure 21.5.2 shows the scheduling of the same configurations for ASHA. Once Trial-1 finishes, we collect the results of two trials (i.e Trial-0 and Trial-1) and immediately promote the better of them (Trial-0) to the next rung level. After Trial-0 finishes on rung 1, there are too few trials there in order to support a further promotion. Hence, we continue with rung 0 and evaluate Trial-3. Once Trial-3 finishes, Trial-2 is still pending. At this point we have 3 trials evaluated on rung 0 and one trial evaluated already on rung 1. Since Trial-3 performs worse than Trial-0 at rung 0, and \(\eta=2\), we cannot promote any new trial yet, and Worker-1 starts Trial-4 from scratch instead. However, once Trial-2 finishes and scores worse than Trial-3, the latter is promoted towards rung 1. Afterwards, we collected 2 evaluations on rung 1, which means we can now promote Trial-0 towards rung 2. At the same time, Worker-1 continues with evaluating new trials (i.e., Trial-5) on rung 0.
21.5.1 Objective Function
We will use Syne Tune with the same objective function as in Section 21.3.
def hpo_objective_lenet_synetune(learning_rate, batch_size, max_epochs):
from d2l import torch as d2l
from syne_tune import Reporter
model = d2l.LeNet(lr=learning_rate, num_classes=10)
trainer = d2l.HPOTrainer(max_epochs=1, num_gpus=1)
data = d2l.FashionMNIST(batch_size=batch_size)
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
report = Reporter()
for epoch in range(1, max_epochs + 1):
if epoch == 1:
# Initialize the state of Trainer
trainer.fit(model=model, data=data)
else:
trainer.fit_epoch()
validation_error = d2l.numpy(trainer.validation_error().cpu())
report(epoch=epoch, validation_error=float(validation_error))We will also use the same configuration space as before:
min_number_of_epochs = 2
max_number_of_epochs = 10
eta = 2
config_space = {
"learning_rate": loguniform(1e-2, 1),
"batch_size": randint(32, 256),
"max_epochs": max_number_of_epochs,
}
initial_config = {
"learning_rate": 0.1,
"batch_size": 128,
}21.5.2 Asynchronous Scheduler
First, we define the number of workers that evaluate trials concurrently. We also need to specify how long we want to run random search, by defining an upper limit on the total wall-clock time.
# Each LeNet trial fits in well under 7 GB of GPU memory, so we can pack
# multiple trials per device. `PythonBackend(rotate_gpus=True)` (the
# default) round-robins trials across detected GPUs and falls back to
# sharing when `n_workers > num_gpus`. Allocate 7 GB per slot — this
# yields 3 slots on a 24 GB card and 4 slots on a 32 GB card after
# driver overhead, e.g. 4×24 GB → 12 slots; 2×32 GB → 8.
import torch
_GB = 1024 ** 3
n_workers = sum(
torch.cuda.get_device_properties(i).total_memory // (7 * _GB)
for i in range(torch.cuda.device_count())
) or 1
max_wallclock_time = 15 * 60 # 15 minutesThe code for running ASHA is a simple variation of what we did for asynchronous random search.
mode = "min"
metric = "validation_error"
resource_attr = "epoch"
scheduler = ASHA(
config_space,
metric=metric,
mode=mode,
points_to_evaluate=[initial_config],
max_resource_attr="max_epochs",
resource_attr=resource_attr,
grace_period=min_number_of_epochs,
reduction_factor=eta,
)max_resource_level = 10, as inferred from config_spaceMaster random_seed = 3937097272Here, metric and resource_attr specify the key names used with the report callback, and max_resource_attr denotes which input to the objective function corresponds to \(r_{\mathrm{max}}\). Moreover, grace_period provides \(r_{\mathrm{min}}\), and reduction_factor is \(\eta\). We can run Syne Tune as before (this will take about 15 minutes):
trial_backend = PythonBackend(
tune_function=hpo_objective_lenet_synetune,
config_space=config_space,
)
stop_criterion = StoppingCriterion(max_wallclock_time=max_wallclock_time)
tuner = Tuner(
trial_backend=trial_backend,
scheduler=scheduler,
stop_criterion=stop_criterion,
n_workers=n_workers,
print_update_interval=int(max_wallclock_time * 0.6),
)
tuner.run()tuning status (last metric is reported)
trial_id status iter learning_rate batch_size max_epochs epoch validation_error worker-time
0 Stopped 2 0.100000 128 10 2 0.900415 12.487616
1 Completed 10 0.021952 205 10 10 0.899944 41.827688
2 Stopped 2 0.023391 233 10 2 0.899977 9.011546
3 Stopped 2 0.226698 176 10 2 0.900031 9.000616
4 Stopped 10 0.090232 176 10 10 0.351475 43.281405
5 Completed 10 0.244815 146 10 10 0.218522 45.709201
6 Stopped 10 0.446042 162 10 10 0.199554 44.254037
7 Stopped 4 0.010802 185 10 4 0.901720 16.155866
8 Stopped 10 0.899590 71 10 10 0.229041 68.296333
9 Completed 10 0.155373 195 10 10 0.287932 41.764440
10 Completed 10 0.045828 89 10 10 0.321459 65.147621
11 Stopped 3 0.093446 176 10 3 0.899942 13.358326
12 Completed 10 0.214394 195 10 10 0.251649 39.342104
13 Stopped 2 0.957507 32 10 2 0.899860 26.079882
14 Completed 10 0.126461 116 10 10 0.262303 44.825703
15 Completed 10 0.246432 224 10 10 0.243651 36.263943
16 Stopped 2 0.013653 66 10 2 0.900037 12.360146
17 Stopped 4 0.017864 118 10 4 0.899993 18.229337
18 Stopped 2 0.011252 236 10 2 0.900796 8.582906
19 Stopped 2 0.019967 70 10 2 0.899933 12.801813
20 Completed 10 0.797668 210 10 10 0.170452 41.484329
21 Completed 10 0.463607 33 10 10 0.132279 111.674271
22 Completed 10 0.079361 61 10 10 0.227214 64.963469
23 Stopped 4 0.018980 95 10 4 0.899861 19.603159
24 Completed 10 0.354178 42 10 10 0.137578 90.663556
25 Stopped 2 0.027259 79 10 2 0.899900 10.849898
26 Stopped 10 0.532411 126 10 10 0.194608 44.455094
27 Stopped 4 0.061722 146 10 4 0.899816 20.419222
28 Completed 10 0.307256 84 10 10 0.183829 72.151050
29 Completed 10 0.184180 49 10 10 0.210154 70.478016
30 Stopped 5 0.011171 204 10 5 0.901961 19.159552
31 Stopped 2 0.045107 62 10 2 0.900438 12.400860
32 InProgress 4 0.144885 41 10 4 0.249487 47.026475
33 Stopped 2 0.245809 225 10 2 0.899753 8.239290
34 InProgress 4 0.851256 38 10 4 0.179725 36.385747
35 InProgress 7 0.778816 127 10 7 0.183696 28.894591
3 trials running, 33 finished (13 until the end), 541.62s wallclock-timereaching max wallclock time (900), stopping there.Stopping trials that may still be running.Tuning finished, results of trials can be found on /home/smola/syne-tune/python-entrypoint-2026-07-19-13-46-40-584--------------------
Resource summary (last result is reported):
trial_id status iter learning_rate batch_size max_epochs epoch validation_error worker-time
0 Stopped 2 0.100000 128 10 2.0 0.900415 12.487616
1 Completed 10 0.021952 205 10 10.0 0.899944 41.827688
2 Stopped 2 0.023391 233 10 2.0 0.899977 9.011546
3 Stopped 2 0.226698 176 10 2.0 0.900031 9.000616
4 Stopped 10 0.090232 176 10 10.0 0.351475 43.281405
5 Completed 10 0.244815 146 10 10.0 0.218522 45.709201
6 Stopped 10 0.446042 162 10 10.0 0.199554 44.254037
7 Stopped 4 0.010802 185 10 4.0 0.901720 16.155866
8 Stopped 10 0.899590 71 10 10.0 0.229041 68.296333
9 Completed 10 0.155373 195 10 10.0 0.287932 41.764440
10 Completed 10 0.045828 89 10 10.0 0.321459 65.147621
11 Stopped 3 0.093446 176 10 3.0 0.899942 13.358326
12 Completed 10 0.214394 195 10 10.0 0.251649 39.342104
13 Stopped 2 0.957507 32 10 2.0 0.899860 26.079882
14 Completed 10 0.126461 116 10 10.0 0.262303 44.825703
15 Completed 10 0.246432 224 10 10.0 0.243651 36.263943
16 Stopped 2 0.013653 66 10 2.0 0.900037 12.360146
17 Stopped 4 0.017864 118 10 4.0 0.899993 18.229337
18 Stopped 2 0.011252 236 10 2.0 0.900796 8.582906
19 Stopped 2 0.019967 70 10 2.0 0.899933 12.801813
20 Completed 10 0.797668 210 10 10.0 0.170452 41.484329
21 Completed 10 0.463607 33 10 10.0 0.132279 111.674271
22 Completed 10 0.079361 61 10 10.0 0.227214 64.963469
23 Stopped 4 0.018980 95 10 4.0 0.899861 19.603159
24 Completed 10 0.354178 42 10 10.0 0.137578 90.663556
25 Stopped 2 0.027259 79 10 2.0 0.899900 10.849898
26 Stopped 10 0.532411 126 10 10.0 0.194608 44.455094
27 Stopped 4 0.061722 146 10 4.0 0.899816 20.419222
28 Completed 10 0.307256 84 10 10.0 0.183829 72.151050
29 Completed 10 0.184180 49 10 10.0 0.210154 70.478016
30 Stopped 5 0.011171 204 10 5.0 0.901961 19.159552
31 Stopped 2 0.045107 62 10 2.0 0.900438 12.400860
32 Completed 10 0.144885 41 10 10.0 0.177095 107.545409
33 Stopped 2 0.245809 225 10 2.0 0.899753 8.239290
34 Completed 10 0.851256 38 10 10.0 0.117126 86.918784
35 Completed 10 0.778816 127 10 10.0 0.163358 41.555937
36 Stopped 2 0.023277 221 10 2.0 0.900148 8.609520
37 Stopped 10 0.656213 82 10 10.0 0.142705 49.178964
38 Stopped 9 0.348940 233 10 9.0 0.259968 34.203501
39 Completed 10 0.421441 250 10 10.0 0.238500 37.586801
40 Stopped 5 0.013137 244 10 5.0 0.899992 19.341023
41 Stopped 2 0.011445 163 10 2.0 0.900497 8.894141
42 Completed 10 0.394467 41 10 10.0 0.137778 86.837596
43 Stopped 2 0.038175 140 10 2.0 0.900397 8.752890
44 Stopped 2 0.069544 158 10 2.0 0.900385 8.897889
45 Stopped 2 0.143401 254 10 2.0 0.900235 8.721123
46 Stopped 4 0.158694 145 10 4.0 0.373492 16.613237
47 Stopped 2 0.043617 126 10 2.0 0.899413 9.097254
48 Stopped 4 0.042216 55 10 4.0 0.392497 27.301295
49 Completed 10 0.329200 160 10 10.0 0.217659 40.404337
50 Stopped 2 0.028384 241 10 2.0 0.900496 7.984386
51 Stopped 2 0.017426 73 10 2.0 0.900006 11.176608
52 Stopped 4 0.109214 122 10 4.0 0.417006 18.057146
53 Stopped 2 0.011410 58 10 2.0 0.899915 14.112154
54 Stopped 2 0.027741 96 10 2.0 0.900298 10.653170
55 Stopped 4 0.198722 191 10 4.0 0.400420 16.728522
56 Stopped 2 0.073597 87 10 2.0 0.900013 10.921521
57 Completed 10 0.289185 219 10 10.0 0.232808 40.402213
58 Stopped 4 0.010721 111 10 4.0 0.901000 18.978153
59 Stopped 3 0.045611 111 10 3.0 0.901000 14.465207
60 Stopped 2 0.018952 210 10 2.0 0.900061 8.634510
61 Stopped 10 0.226332 48 10 10.0 0.152113 69.217495
62 Completed 10 0.458785 154 10 10.0 0.184410 38.589711
63 Stopped 10 0.608629 124 10 10.0 0.153649 40.232968
64 InProgress 8 0.764240 221 10 8.0 0.198532 29.016161
65 InProgress 8 0.619694 196 10 8.0 0.217622 29.298571
66 Stopped 2 0.022491 253 10 2.0 0.899848 7.448091
67 InProgress 0 0.553540 218 10 - - -
3 trials running, 65 finished (21 until the end), 903.53s wallclock-time
validation_error: best 0.11712592840194702 for trial-id 34
--------------------Note that we are running a variant of ASHA where underperforming trials are stopped early. This is different to our implementation in Section 21.4.1, where each training job is started with a fixed max_epochs. In the latter case, a well-performing trial which reaches the full 10 epochs, first needs to train 1, then 2, then 4, then 8 epochs, each time starting from scratch. This type of pause-and-resume scheduling can be implemented efficiently by checkpointing the training state after each epoch, but we avoid this extra complexity here. After the experiment has finished, we can retrieve and plot results.
d2l.set_figsize()
e = load_experiment(tuner.name)
e.plot()
21.5.3 Visualize the Optimization Process
Once more, we visualize the learning curves of every trial (each color in the plot represents a trial). Compare this to asynchronous random search in Section 21.3. As we have seen for successive halving in Section 21.4, most of the trials are stopped at 1 or 2 epochs (\(r_{\mathrm{min}}\) or \(\eta * r_{\mathrm{min}}\)). However, trials do not stop at the same point, because they require different amount of time per epoch. If we ran standard successive halving instead of ASHA, we would need to synchronize our workers, before we can promote configurations to the next rung level.
d2l.set_figsize([6, 2.5])
results = e.results
for trial_id in results.trial_id.unique():
df = results[results["trial_id"] == trial_id]
d2l.plt.plot(
df["st_tuner_time"],
df["validation_error"],
marker="o"
)
d2l.plt.xlabel("wall-clock time")
d2l.plt.ylabel("objective function")Text(0, 0.5, 'objective function')
21.5.4 Summary
Compared to random search, successive halving is not quite as trivial to run in an asynchronous distributed setting. To avoid synchronization points, we promote configurations as quickly as possible to the next rung level, even if this means promoting some wrong ones. In practice, this usually does not hurt much, and the gains of asynchronous versus synchronous scheduling are usually much higher than the loss of the suboptimal decision making.