bfbrain package

Submodules

bfbrain.AL_Metrics module

This module contains code for various performance metrics which BFBrain can track over the course of active learning.

bfbrain.AL_Metrics.process_score_fn(score_fn, name)

A utility function which translates a string specifying one of the predefined acquisition scoring functions into the corresponding numerical method.

Parameters:

• score_fn ({'BALD', 'QBDC', 'random', 'MaxEntropy', 'variation_ratios', 'predictive_variance'} or callable.) – If this function is a callable, it must have the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32)

• name (str, optional) –

  If specified, this name is returned unaltered. Otherwise,

  a name will be automatically generated based on score_fn.

Returns:

• callable – A valid score_fn to be used in various performance metrics.

• str – A string which will be used to generate a name for an ALMetric object.

class bfbrain.AL_Metrics.ALMetric(sc_type, name)

Bases: ABC

A generic abstract class for computing and recording performance metrics for active learning. All performance metrics in BFBrain inherit from this class.

status_history

A list which contains a record, for each round of active learning, for whichever metric the subclass will measure. The entries of status_history may be, depending on the subclass, virtually any kind of data or data structure, as long as the elements are picklable.

Type:

list

sc_type

A string which denotes what type of metric the ALMetric object is, since different metrics are recorded at different points in the active learning loop. If sc_type is ‘val’, this metric is computed using a validation data set immediately after each active learning round completes. If sc_type is ‘train’, this metric is computed immediately after new training data is generated in the active learning loop, but before the neural network’s weights are reset and training commences. It is evaluated using the newly-generated training data. If sc_type is ‘pool’, this metric is computed using the pool of candidate points from which new training samples are drawn at each iteration. It is computed immediately after the new training data is selected from the pool. If sc_type is model, the a metric is computed without reference to any data set (validation, training, or pool) present in the active learning loop, at the end of each active learning iteration. The only implemented metrics which have sc_type ‘model’ measure predictive stability on some specified unlabelled set of points, namely UnlabelledAgreement and UnlabelledDeltaFScore, but the possibility remains that different sorts of metrics in this class, for example the one based on error stability computed directly from the neural network weights discussed in arXiv:2104.01836, may be desirable for a user to implement.

Type:

{‘val’, ‘train’, ‘pool’, ‘model’}

name

A string which denotes a name for this metric. In a list of metrics passed to a BFBLearner class, the names of each member of the list should be unique.

Type:

str

Parameters:

• sc_type ({'val', 'train', 'pool', 'model'}) –

• name (str) –

record_score(*args)

Appends the latest value for the performance metric to the status_history object. This method calls an abstract method “performance_check” which will turn whatever input is specified in the method into the metric the object is supposed to track. The method performance_check, and therefore the arguments going into this method, will vary depending on the specific subclass of ALMetric.

print_status(file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

A method which prints the last entry in status_history to a file (or the console). Uses the method perf_message (which is often overwritten in the child class) to identify the metric being printed and separates status_history elements that are tuples into different printout lines, for clarity.

perf_message()

A method which prints out a message that is helpful in identifying what metric is being reported when a user calls print_status. Often overwritten in a child class.

abstract performance_check(*args)

An abstract class which takes some arguments (depending on the type of performance metric) and computes the quantity or quantities that the performance metric is supposed to track. This method is called by record_score and its results are appended to the status_history attribute.

get_metric(*args)

A function which reduces the status_history object to a list of single numbers (usually some sort of figure of merit) in the event that the members of status_history are a list or a tuple. By default, it simply returns the full status_history list and must be overwritten in subclasses which have lists or tuples as entries in status_history.

Parameters:

*args (Any) – Some overwritten versions of this class can accept optional arguments, although the method does not in the parent class.

Return type:

A NumPy array featuring information from status_history for plotting.

reset_data()

A function which resets the metric data entirely. In some subclasses, this must be overloaded to properly reset the class.

get_legend(*args)

Returns a legend for a plot of the metric given by plot_metric. Often must be overwritten in subclasses.

Parameters:

*args (Any) – Some overwritten versions of this class can accept optional arguments, although the method does not in the parent class. Must take the same arguments as get_metric.

Returns:

A list of strings which are usable to specify a legend in matplotlib.

Return type:

list of strings

plot_metric(filepath=None, **kwargs)

Plots the performance metric as a function of the number of active learning iterations.

Parameters:

• filepath (str, optional) – If this argument is specified, then the plot of the metric will be saved as a .png file in the directory with the name given by filepath.

• **kwargs (dict, optional) – Many subclasses of ALMetric have get_metric and get_legend methods which take some keyword arguments– these can be specified when calling plot_metric.

class bfbrain.AL_Metrics.ModelMetric(name)

Bases: ALMetric

An abstract class for handling metrics which depend only on the BFBLearner object’s model, plus some consistent internal information. This class can be used as a “catch-all” for metrics which don’t fit neatly into other categories– for example, we use it in UnlabelledPredsMetric and its child classes to track the predictions of the model on some unlabelled set of inputs.

abstract performance_check(model)

The performance_check method now specifies the arguments that a class inheriting from TrainMetric should use.

class bfbrain.AL_Metrics.UnlabelledPredsMetric(lams, name, batch_size=200000)

Bases: ModelMetric

An abstract class for handling metrics which go by the predictions of the model on some unlabelled set of quartic coefficients.

lams

A 2-D NumPy array representing sets of quartic potential coefficients. This will be an unlabelled set of points the model will make predictions on.

Type:

np.array(np.float32, np.float32)

ds

A Tensorflow dataset generated from lams.

Type:

tf.data.Dataset

batch_size

The maximum size of batches of lams that will be transferred to the GPU and computed with at one time.

Type:

int, default=200000

name

The unique identifier for the metric in the list of metrics traced by BFBLearner.

Type:

str

Parameters:

• lams (np.array(np.float32, np.float32)) – A 2-D NumPy array representing sets of quartic potential coefficients. This will be an unlabelled set of points the model will make predictions on.

• name (str) – The name will provide a unique identifier for the metric in the list of metrics tracked by BFBLearner– this identifier will be ‘model_’+name.

• batch_size (int, default=200000) – The maximum size of batches of lams that will be transferred to the GPU and computed with at one time.

abstract performance_check(model)

The performance_check method now specifies the arguments that a class inheriting from TrainMetric should use.

class bfbrain.AL_Metrics.UnlabelledAgreement(lams, name='agreement', batch_size=200000, n_trials=100)

Bases: UnlabelledPredsMetric

A metric which computes agreement (Cohen’s kappa) among the model between successive iterations of active learning on a specified set of unlabelled points.

status_history

The entries of this status_history object will be Cohen’s kappa between successive iterations of active learning on lams.

Type:

list of floats

old_preds

The previous model’s predictions on lams. Preserved to compare to the current model.

Type:

np.array(np.float32)

lams

A 2-D NumPy array representing sets of quartic potential coefficients. This will be an unlabelled set of points the model will make predictions on.

Type:

np.array(np.float32, np.float32)

ds

A Tensorflow dataset generated from lams.

Type:

tf.data.Dataset

batch_size

The maximum size of batches of lams that will be transferred to the GPU and computed with at one time.

Type:

int, default=200000

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default this will be ‘model_agreement’

Type:

str

n_trials

The number of forward passes through the network to get the predictions from Monte Carlo dropout.

Type:

int, default=100

Parameters:

• lams (np.array(np.float32, np.float32)) – A 2-D NumPy array representing sets of quartic potential coefficients. This will be an unlabelled set of points the model will make predictions on.

• name (str, default='agreement') – The name will provide a unique identifier for the metric in the list of metrics tracked by BFBLearner– this identifier will be ‘model_’+name.

• batch_size (int, default=200000) – The maximum size of batches of lams that will be transferred to the GPU and computed with at one time.

• n_trials (int, default=100) – The number of forward passes through the network to get the predictions from Monte Carlo dropout.

performance_check(model)

Computes Cohen’s kappa for the classifier between successive active learning iterations, on the unlabelled quartic coefficients lams. If there is no previous model (i.e., no active learning has been done), returns 0. and saves the current model’s predictions as old_preds.

get_metric()

Simply returns Cohen’s kappa as a function of the number of active learning iterations.

reset_data()

Resets the data in the metric.

class bfbrain.AL_Metrics.UnlabelledDeltaF(lams, name='delta_F', batch_size=200000, n_trials=100)

Bases: UnlabelledPredsMetric

A metric which computes the estimated change in F score on a specified unlabelled set of points for the model between successive iterations of active learning on a specified set of unlabelled points, based on the methodology of arXiv:cs/1901.09118.

status_history

The entries of this status_history object will be the estimated change in F score between successive iterations of active learning on lams.

Type:

list of floats

old_preds

The previous model’s predictions on lams. Preserved to compare to the current model.

Type:

np.array(np.float32)

lams

A 2-D NumPy array representing sets of quartic potential coefficients. This will be an unlabelled set of points the model will make predictions on.

Type:

np.array(np.float32, np.float32)

ds

A Tensorflow dataset generated from lams.

Type:

tf.data.Dataset

batch_size

The maximum size of batches of lams that will be transferred to the GPU and computed with at one time.

Type:

int, default=200000

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default this will be ‘model_delta_F’

Type:

str

n_trials

The number of forward passes through the network to get the predictions from Monte Carlo dropout.

Type:

int, default=100

Parameters:

• lams (np.array(np.float32, np.float32)) – A 2-D NumPy array representing sets of quartic potential coefficients. This will be an unlabelled set of points the model will make predictions on.

• name (str, default='delta_F') – The name will provide a unique identifier for the metric in the list of metrics tracked by BFBLearner– this identifier will be ‘model_’+name.

• batch_size (int, default=200000) – The maximum size of batches of lams that will be transferred to the GPU and computed with at one time.

• n_trials (int, default=100) – The number of forward passes through the network to get the predictions from Monte Carlo dropout.

performance_check(model)

Computes the estimated change in F score for the classifier between successive active learning iterations, on the unlabelled quartic coefficients lams. If there is no previous model (i.e., no active learning has been done), returns np.inf and saves the current model’s predictions as old_preds.

get_metric()

Simply returns the change in F score as a function of the number of active learning iterations.

reset_data()

Resets the data in the metric.

class bfbrain.AL_Metrics.ValidationMetric(name)

Bases: ALMetric

An abstract class for handling metrics which measure performance of the model on a validation set. This class exists primarily to remind the user that any validation-set-based performance metrics must have their performance_check method take the inputs (tf.keras.Model, tf.data.Dataset)

abstract performance_check(model, ds)

The performance_check method now specifies the arguments that a class inheriting from ValidationMetric should use.

Parameters:

• model (tf.keras.Model) – The model in a BFBLearner object.

• ds (tf.data.Dataset) – A Tensorflow dataset representing the labelled validation set.

class bfbrain.AL_Metrics.TrainMetric(name)

Bases: ALMetric

An abstract class for handling metrics which measure predictions of the model on newly-added training data. This class exists primarily to remind the user that any training-set-based performance metrics must have their performance_check method take the inputs (tf.keras.Model, tf.Tensor)

abstract performance_check(model, lams, labels)

The performance_check method now specifies the arguments that a class inheriting from TrainMetric should use.

Parameters:

• model (tf.keras.Model) – The model in a BFBLearner object.

• lams (tf.Tensor(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients. This will be all of the points that have been newly added to the training set in a given round of active learning.

• labels (tf.Tensor(bool)) – A 1-D Tensorflow tensor of booleans representing the labels of the new training data points. The ith element of this tensor is True if the ith row of lams represents a set of quartic potential coefficients that the oracle has labelled as bounded-from-below, False otherwise.

class bfbrain.AL_Metrics.PoolMetric(name)

Bases: ALMetric

An abstract class for handling metrics which measure predictions of the model on the pools of candidate points from which new training data is drawn. This class exists primarily to ensure that these metrics have additional abstract methods which much be specified to implement a class of this sort.

batch_scores

Because the pool of candidate points are generated in discrete manageable batches, the metrics are computed over each individual batch and then combined, the precise manner of which depends on the specific metric in question. However, all pool metrics must have this attribute to act as temporary storage of the individual batch results before they can be combined.

Type:

list

abstract record_batch(*args)

An abstract method which will record an individual batch’s results to batch_scores, in a manner that must be specified in a subclass.

abstract record_score()

The concrete record_score method of the ALMetric class must be overwritten with an abstract version, which must in turn be specified in subclasses.

class bfbrain.AL_Metrics.PoolMetricReduction(name, red=['mean', 'min', 'max'])

Bases: PoolMetric

An abstract class which inherits from PoolMetric, but features methods to automatically take the mean/min/max of the scores determined in record_batch. This is an abstract class which shouldn’t be instantiated directly, but allows for rapid prototyping of a variety of pool metrics.

reduction

This is the reduction that is performed on individual batches, and finally, among the results for all batches, to produce the entries in status_history. If a list of reductions are applied, then the elements of status_history will be lists with each element being a different reduction being applied to the pool score.

Type:

a list containing elements of {np.mean, np.min, np.max}

red_name

This list of strings will contain the same information as reduction, but is used for labelling purposes.

Type:

a list containing elements of {‘mean’, ‘min’, ‘max’}

Parameters:

• name (str) – A unique identifier for the metric in the list of metrics in a BFBLearner object.

• red ({'mean', 'min', 'max'} or a list of those values) – This argument specifies the reduction that is performed on individual batches, and finally, among the results for all batches, to produce the entries in status_history. If a list of reductions are applied, then the elements of status_history will be lists with each element being a different reduction being applied to the pool score.

record_batch(*args)

Records a score generated by performance_check (which must be specified in a subclass) for an individual batch in the pool of candidate points to the batch_scores list.

record_score()

Combines the metrics computed for each batch into a single status_history entry, and append that entry to status_history.

perf_message()

The perf_message method labels any printed output of the metric.

get_metric(reduction=None)

In this subclass, get_metric can take a keyword argument.

Parameters:

reduction (str in self.red_name, optional) – If specified, get_metric will only return the values corresponding to the specified reduction. If not specified, get_metric will return the status_history object in its entirety.

Returns:

Represents some plottable set of values from status_history.

Return type:

np.array

reset_data()

Reset the data in status_history.

get_legend(reduction=None)

In this subclass, get_legend can take a keyword argument.

Parameters:

reduction (str in self.red_name, optional) – get_legend will return a legend consistent with the get_metric result with the same reduction argument passed.

Returns:

An argument to specify a legend in matplotlib.

Return type:

list of str

class bfbrain.AL_Metrics.PoolDeltaF(name='delta_F')

Bases: PoolMetric

A metric for keeping track of the estimated change in F score for successive iterations of the active learning algorithm on unlabelled data, based on arXiv:cs/1901.09118. Each time active learning produces a new unlabelled pool of candidate points to draw training examples from, this metric computes the model’s predicted labels for all of these points, and stores both the pool of points and the predictions. Then, after another round of active learning, the metric computes the NEW model’s predicted labels on the stored pool of points. Then, the two sets of predictions are compared and the estimated change in F score over the pool distribution is computed from the level of agreement between the two sets of predictions, following the procedure outlined in arXiv:cs/1901.09118. Predictions are based on Monte Carlo dropout with 100 forward passes through the neural network.

status_history

A list which contains a record, for each round of active learning, of the estimated change in F score.

Type:

list of floats

name

A string which denotes a name for this metric. In a list of metrics passed to a BFBLearner class, the names of each member of the list should be unique.

Type:

str, default=’delta_F’

old_pool

A list of 2-D NumPy arrays, each of which represents a batch of points generated as part of the pool of candidate points in active learning. This array stores the points that made up the PREVIOUS round’s pool of points, so that the current model can make predictions on them.

Type:

list of np.array(np.float32, np.float32)

new_pool

A list of 2-D NumPy arrays, each of which represents a batch of points generated as part of the pool of candidate points in active learning. This array stores the points that made up the CURRENT round’s pool of points, so that the current model can make predictions on them. After recording the status_history value for this metric, new_pool’s values are transferred to old_pool, and then new_pool is cleared.

Type:

list of np.array(np.float32, np.float32)

old_preds

A list to contain all of the PREVIOUS model’s predictions on old_pool.

Type:

list of np.array(np.float32)

new_preds

A list to contain all of the CURRENT model’s predictions on old_pool.

Type:

list of np.array(np.float32)

newer_preds

A list to contain all of the CURRENT model’s predictions on new_pool. After recording the status_history value for this metric, newer_preds values are transferred to old_preds, and then newer_preds and new_preds are both cleared.

Type:

list of np.array(np.float32)

old_pool_iter

An iterator over old_pool. Replaced whenever old_pool is overwritten.

Type:

iter

record_batch(model, L)

Records newly-generated pool points and the current model’s prediction on them. After the first active learning iteration, also records the current model’s predictions on the corresponding element of old_pool, since after the first active learning iteration the old_pool and new_pool will always have the same number of elements.

Parameters:

• model (tf.keras.model) – The current Tensorflow model of a BFBLearner object.

• L (np.array(np.float32, np.float32)) – A 2-D NumPy array representing a batch of pool points proposed to the neural network as possible training points.

performance_check(model, L)

Computes the predictions of the neural network on an input batch of pool points, and returns a tuple of the predictions and the pool points.

Parameters:

• model (tf.keras.model) – The current Tensorflow model of a BFBLearner object.

• L (np.array(np.float32, np.float32)) – A 2-D NumPy array representing a batch of pool points proposed to the neural network as possible training points.

Returns:

• np.array(np.float32) – A 1-D NumPy array of model predictions on L

• np.array(np.float32, np.float32) – The array L

record_score()

Combines the predictions made on individual batches in order to produce an estimate of the change in F score on old_pool, and appends this estimate onto status_history. Then, overwrites old_pool with new_pool, old_preds with newer_preds, and then resets new_pool, new_preds, and newer_preds.

Returns:

The last element of status_history.

Return type:

float

reset_data()

Clears all data in the metric.

class bfbrain.AL_Metrics.ModelEvaluation(name='accuracy')

Bases: ValidationMetric

A metric that simply records the model.evaluate() method on a labelled Tensorflow dataset (the validation set in our context). BFBrain’s call to Tensorflow’s evaluate method keeps track of the model accuracy, false positives, and false negatives via evaluate(). Note that this method does NOT use Monte Carlo dropout to compute these quantities, but instead approximates the mean of Monte Carlo dropout via a single pass through the network with no dropout (and all model weights divided by 1 - <dropout probability>).

status_history

The entries of status_history here will be lists of the form [<binary accuracy>, <false positives>, <false negatives>], evaluated over the validation set.

Type:

list of lists of np.float32

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default, this will be ‘val_accuracy’.

Type:

str

Parameters:

name (str, default='accuracy') – The name will provide a unique identifier for the metric in the list of metrics tracked by BFBLearner– this identifier will be ‘val_’+name.

performance_check(model, ds)

This performance_check simply calls Tensorflow’s model.evaluate() method, and ignores the first term (the loss)

perf_message()

The perf_message method labels any printed output of the metric.

get_metric()

For plotting, this metric simply requests the binary accuracy over the active learning iterations.

class bfbrain.AL_Metrics.MCModelEvaluation(name='MC_accuracy')

Bases: ValidationMetric

A metric that records the same data as ModelEvaluation on a labelled Tensorflow dataset (the validation set in our context), but using Monte Carlo dropout with 100 forward passes through the neural network. Otherwise functions identically to ModelEvaluation.

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default, this will be ‘val_MC_accuracy’.

Type:

str

Parameters:

name (str, default='MC_accuracy') – The name will provide a unique identifier for the metric in the list of metrics tracked by BFBLearner– this identifier will be ‘val_’+name.

performance_check(model, ds)

This performance_check finds the binary accuracy, false positives, and false negatives over a validation data set.

perf_message()

The perf_message method labels any printed output of the metric.

get_metric()

A function which reduces the status_history object to a list of single numbers (usually some sort of figure of merit) in the event that the members of status_history are a list or a tuple. By default, it simply returns the full status_history list and must be overwritten in subclasses which have lists or tuples as entries in status_history.

Parameters:

*args (Any) – Some overwritten versions of this class can accept optional arguments, although the method does not in the parent class.

Return type:

A NumPy array featuring information from status_history for plotting.

class bfbrain.AL_Metrics.DecisionBoundaryScore(tol_dist, name='combined_false_score')

Bases: ValidationMetric

A metric which records a “decision boundary score”– for each point that the (non-MC-dropout) neural network classifies incorrectly in a validation set, this method uses gradient ascent/descent to determine the angular distance on the hypersphere of quartic coeffecients to the decision boundary. Reports the results of the mean, standard deviation, and max of these scores in radians for both false positives and false negatives, as well as the number of points in both groups which exceed some input number of radians distance from the decision boundary. This metric can be extremely computationally intensive, and generally can reflect the deterministic forward pass’s tendency to occasionally be incorrect and very overconfident. However, if a user is insistent on only using a single forward pass of the neural network to evaluate a network, this method enables them to be somewhat confident that any points that are mislabelled will be close in parameter space to points which are correctly labelled.

status_history

The entries of status_history here will contain a tuple of two lists of the form [<mean>, <std>, <max>, # > tol_dist radians], the first for false positives and the second for false negatives. The mean, standard deviation, and max values are computed from the angular distance (in radians) of the incorrectly classified points to points along the decision boundary. The final entry is the number of points for which this distance is greater than some user-specified cutoff, tol_dist.

Type:

list of tuples of lists of np.float32

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default, this will be ‘val_combined_false_score’.

Type:

str

tol_dist

The maximum angular distance of an incorrectly-classified point to the decision boundary that the user considers acceptable. For small (O(0.01)) values of this angle, it roughly corresponds to the fractional degree of correction of the quartic coefficients to reach the decision boundary– so an angular deformation of 0.01 represents approximately a 1% correction to the quartic coupling coefficients.

Type:

float

Parameters:

• tol_dist (float) – The maximum angular distance of an incorrectly-classified point to the decision boundary that the user considers acceptable. For small (O(0.01)) values of this angle, it roughly corresponds to the fractional degree of correction of the quartic coefficients to reach the decision boundary– so an angular deformation of 0.01 represents approximately a 1% correction to the quartic coupling coefficients.

• name (str, default='combined_false_score') – The name will provide a unique identifier for the metric in the list of metrics tracked by BFBLearner– this identifier will be ‘val_’+name.

performance_check(model, ds)

Computes the angular distances between incorrectly classified points and the decision boundary.

perf_message()

The perf_message method labels any printed output of the metric.

get_metric()

Returns the number of false positives and false negatives greater than tol_dist radians from the decision boundary (tracked separately) for plotting over the course of active learning.

class bfbrain.AL_Metrics.ValidationConfusionMatrix(score_fn='BALD', name=None, quantiles=[0.95, 1.0], n_trials=None)

Bases: ValidationMetric

A metric that finds the elements of the confusion matrix (correctly labelled positives, false positives, correctly labelled negatives, false negatives) for the validation set. Also calculates the confusion matrix with points which score higher than specified quantiles on some specified uncertainty metric, evaluated over all points which have the same predicted classification, omitted from the validation set. This metric in turn has all the information necessary for the extraction of binary classifier quality metrics such as precision, recall, or F score.

status_history

The elements status_history here are tuples of the form (<true positives>, <false positives>, <true negatives>, <false negatives>), where each element of the tuple is a list of length equal to the length of the attribute quantiles. Each element of the lists are the values of that observable assuming that we only include points with an uncertainty score (given by score_fn) less than or equal to the corresponding quantile (evaluated for all points of the same predicted class) in quantiles.

Type:

list of tuples of lists of ints

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default this will be ‘val_<score_fn>_confusion’

Type:

str

score_fn

A callable of the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32). The pre-implemented functions for different uncertainty metrics can be specified in the constructor by using any of ‘BALD’ (mutual information), ‘MaxEntropy’ (Shannon entropy), ‘variation_ratios’ (variation ratios), ‘predictive_variance’ (variance of the prediction distribution), or ‘QBDC’ (score*(1-score), where score is the Monte Carlo dropout-evaluated prediction for an input)

Type:

callable

tf_score_fn

Tensorflow jit-compiled version of score_fn.

Type:

jit-compiled callable

n_trials

Type:

int or None

Parameters:

• score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Specifies the score function that the metric will apply to the pool of candidate points. If a callable (corresponding to a custom score function) is used, it must have the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether or not n_trials is specified.

• name (str, optional) – Allows for a custom name of the metric. If this argument is not specified, a name will be automatically generated as ‘val_<score_fn>_confusion’.

• quantiles (list of floats, default=[0.95, 1.]) – The uncertainty quantiles for which the metric is tracked (see the status_history documentation)

• n_trials (int, optional) – For score_fn arguments that take a n_trials argument, which includes every pre-implemented score_fn except ‘random’, this argument can be specified here. If it is not specified, the default value for the given score_fn is used.

performance_check(model, ds)

Finds the confusion matrix (true positives, false positives, true negatives, false negatives) for the model over the validation set.

perf_message()

The perf_message method labels any printed output of the metric.

get_metric(quantile=None)

In this subclass, get_metric can take a keyword argument.

Parameters:

quantile (float in self.quantiles, optional) – If specified, get_metric will only return the values corresponding to the false positives and false negatives found with uncertainty scores less than or equal to the given quantile. Otherwise, all false positives and false negatives for all quantiles will be returned for plotting.

Returns:

Represents some plottable set of values from status_history.

Return type:

np.array

get_legend(quantile=None)

In this subclass, get_metric can take a keyword argument.

Parameters:

quantile (float in self.quantiles, optional) – The method will return a legend appropriate for plotting get_metric, when get_metric is given the same arguments.

Returns:

A list of strings representing a plot legend for matplotlib.

Return type:

list of str

class bfbrain.AL_Metrics.ValidationFScore(score_fn='BALD', name=None, quantiles=[0.95, 1.0], n_trials=None)

Bases: ValidationConfusionMatrix

A metric that finds the precision, recall, and F score for the validation set. Also calculates the confusion matrix with points which score higher than specified quantiles on some specified uncertainty metric, evaluated over all points which have the same predicted classification, omitted from the validation set. This metric in turn has all the information necessary for the extraction of binary classifier quality metrics such as precision, recall, or F score.

status_history

The elements of the status_history object here are lists of the form [<precision>, <recall>, <F score>], where each element of the tuple is a list of length equal to the length of the attribute quantiles. Each element of the lists are the values of that observable assuming that we only include points with an uncertainty score (given by score_fn) less than or equal to the corresponding quantile (evaluated for all points of the same predicted class) in quantiles.

Type:

list of lists of lists of floats

name

The unique identifier for the metric in the list of metrics traced by BFBLearner. By default this will be ‘val_<score_fn>_fscore’

Type:

str

score_fn

A callable of the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32). The pre-implemented functions for different uncertainty metrics can be specified in the constructor by using any of ‘BALD’ (mutual information), ‘MaxEntropy’ (Shannon entropy), ‘variation_ratios’ (variation ratios), ‘predictive_variance’ (variance of the prediction distribution), or ‘QBDC’ (score*(1-score), where score is the Monte Carlo dropout-evaluated prediction for an input)

Type:

callable

tf_score_fn

Tensorflow jit-compiled version of score_fn.

Type:

jit-compiled callable

n_trials

Type:

int or None

Parameters:

• score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Specifies the score function that the metric will apply to the pool of candidate points. If a callable (corresponding to a custom score function) is used, it must have the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether or not n_trials is specified.

• name (str, optional) – Allows for a custom name of the metric. If this argument is not specified, a name will be automatically generated as ‘val_<score_fn>_fscore’.

• quantiles (list of floats, default=[0.95, 1.]) – The uncertainty quantiles for which the metric is tracked (see the status_history documentation)

• n_trials (int, optional) – For score_fn arguments that take a n_trials argument, which includes every pre-implemented score_fn except ‘random’, this argument can be specified here. If it is not specified, the default value for the given score_fn is used.

performance_check(model, ds)

Computes the precision, recall, and F score over the validation set.

perf_message()

The perf_message method labels any printed output of the metric.

get_metric(quantile=None)

In this subclass, get_metric can take a keyword argument.

Parameters:

quantile (float in self.quantiles, optional) – If specified, get_metric will only return the values corresponding to the F score found with uncertainty scores less than or equal to the given quantile. Otherwise, all F scores for all quantiles will be returned for plotting.

Returns:

Represents some plottable set of values from status_history.

Return type:

np.array

get_legend(quantile=None)

In this subclass, get_metric can take a keyword argument.

Parameters:

quantile (float in self.quantiles, optional) – The method will return a legend appropriate for plotting get_metric, when get_metric is given the same arguments.

Returns:

A list of strings representing a plot legend for matplotlib.

Return type:

list of str

class bfbrain.AL_Metrics.PoolScore(score_fn='BALD', name=None, reduction=['mean', 'min', 'max'], n_trials=None)

Bases: PoolMetricReduction

A metric which applies the function score_fn to the pool of candidate points at every active learning iteration, before the model is trained on any new data drawn from the pool. Evaluates score_fn on the pool points and records specified reductions of these scores.

status_history

Each entry in this status_history object has entries corresponding to the score_fn results applied to each active learning iteration’s pool of candidate points, with reduction(s) specified in the constructor. If the constructor specifies multiple reductions, each entry is a list with each value’s reduction (so an entry will be [<mean>, <max>, <min>], for example).

Type:

list of lists of floats (or np.float32)

name

A string which denotes a name for this metric. In a list of metrics passed to a BFBLearner class, the names of each member of the list should be unique. By default will be ‘pool_<score_fn>’

Type:

str

score_fn

A callable of the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32). The pre-implemented functions for different uncertainty metrics can be specified in the constructor by using any of ‘BALD’ (mutual information), ‘MaxEntropy’ (Shannon entropy), ‘variation_ratios’ (variation ratios), ‘predictive_variance’ (variance of the prediction distribution), or ‘QBDC’ (score*(1-score), where score is the Monte Carlo dropout-evaluated prediction for an input)

Type:

callable

tf_score_fn

Tensorflow jit-compiled version of score_fn.

Type:

jit-compiled callable

n_trials

Type:

int or None

Parameters:

• score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Specifies the score function that the metric will apply to the pool of candidate points. If a callable (corresponding to a custom score function) is used, it must have the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether or not n_trials is specified.

• name (str, optional) – Allows for a custom name of the metric. If this argument is not specified, a name will be automatically generated as ‘pool_<score_fn>’.

• reduction ({'mean', 'min', 'max'} or a list of these values, default=['mean','min','max']) – Specifies what reductions should be done on the scores computed for the pool candidate points. If a list is specified, all reductions in the list are computed.

• n_trials (int, optional) – For score_fn arguments that take a n_trials argument, which includes every pre-implemented score_fn except ‘random’, this argument can be specified here. If it is not specified, the default value for the given score_fn is used.

performance_check(*args)

performance_check for this metric records the score function evaluated on a pool of candidate points, subject to the reduction(s) specified in the constructor.

Parameters:

*args ((tf.keras.Model, tf.tensor(tf.float32, tf.float32)) or tf.tensor(tf.float32)) – If the class’s score_fn were already evaluated as part of active learning (namely, if the score_fn corresponds to the one used as the acquisition function in active learning), then this method can take an already-evaluated Tensorflow tensor consisting of a list of scores. If not, then we can pass the arguments appropriate for score_fn in order to evaluate the results directly.

Returns:

A list of the specified reductions in the score function for a batch of pool candidate points. These are then combined into a single entry into status_history using the methods in the parent class.

Return type:

list of np.float32

class bfbrain.AL_Metrics.NewDataScore(score_fn='BALD', name=None, red=['mean', 'min', 'max'], n_trials=None)

Bases: TrainMetric

A metric which applies the function score_fn to the set of points that are added to the training set at every active learning iteration, before the model is trained on the new data. Evaluates score_fn on these points and records specified reductions of these scores.

status_history

Each entry in this status_history object has entries corresponding to the score_fn results applied to each active learning iteration’s new training data, with reduction(s) specified in the constructor. If the constructor specifies multiple reductions, each entry is a list with each value’s reduction (so an entry will be [<mean>, <max>, <min>], for example).

Type:

list of lists of floats (or np.float32)

name

A string which denotes a name for this metric. In a list of metrics passed to a BFBLearner class, the names of each member of the list should be unique. By default will be ‘train_<score_fn>’

Type:

str

score_fn

A callable of the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32). The pre-implemented functions for different uncertainty metrics can be specified in the constructor by using any of ‘BALD’ (mutual information), ‘MaxEntropy’ (Shannon entropy), ‘variation_ratios’ (variation ratios), ‘predictive_variance’ (variance of the prediction distribution), or ‘QBDC’ (score*(1-score), where score is the Monte Carlo dropout-evaluated prediction for an input)

Type:

callable

tf_score_fn

Tensorflow jit-compiled version of score_fn.

Type:

jit-compiled callable

n_trials

Type:

int or None

reduction

This is the reduction that is performed on the scores to produce the entries in status_history. If a list of reductions are applied, then the elements of status_history will be lists with each element being a different reduction being applied to the scores.

Type:

a list containing elements of {np.mean, np.min, np.max}

red_name

This list of strings will contain the same information as reduction, but is used for labelling purposes.

Type:

a list containing elements of {‘mean’, ‘min’, ‘max’}

Parameters:

• score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Specifies the score function that the metric will apply to the pool of candidate points. If a callable (corresponding to a custom score function) is used, it must have the signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether or not n_trials is specified.

• name (str, optional) – Allows for a custom name of the metric. If this argument is not specified, a name will be automatically generated as ‘train_<score_fn>’.

• red ({'mean', 'min', 'max'} or a list of these values, default=['mean','min','max']) – Specifies what reductions should be done on the scores computed for the new training data. If a list is specified, all reductions in the list are computed.

• n_trials (int, optional) – For score_fn arguments that take a n_trials argument, which includes every pre-implemented score_fn except ‘random’, this argument can be specified here. If it is not specified, the default value for the given score_fn is used.

performance_check(model, lams, labels)

Computes score_fn on the points added to the training set with each active learning iteration, and then records the specified reductions.

perf_message()

The perf_message method labels any printed output of the metric.

get_metric(reduction=None)

In this subclass, get_metric can take a keyword argument.

Parameters:

reduction (str in self.red_name, optional) – If specified, get_metric will only return the values corresponding to the specified reduction. If not specified, get_metric will return the status_history object in its entirety.

Returns:

Represents some plottable set of values from status_history.

Return type:

np.array

get_legend(reduction=None)

In this subclass, get_legend can take a keyword argument.

Parameters:

reduction (str in self.red_name, optional) – get_legend will return a legend consistent with the get_metric result with the same reduction argument passed.

Returns:

An argument to specify a legend in matplotlib.

Return type:

list of str

class bfbrain.AL_Metrics.StoppingCondition(metric_name, metric_func)

Bases: object

A generic class for implementing early stopping conditions for active learning. A StoppingCondition object is called each round immediately after the metric it follows is evaluated. Then, if the call returns True, the active learning loop is terminated.

metric_name

Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track.

Type:

str

metric_func

A callable which takes a metric and an (optional) integer index as input and returns True if the stopping condition has been met, and False otherwise. Note that if one wishes to use the method find_stopping_index, the metric_func function MUST be capable of accommodating the optional integer argument. If this argument won’t be called, it’s acceptable to use an existing argument.

Type:

callable

Parameters:

• metric_name (str) – Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track

• metric_func (callable) – A callable which takes a metric and an (optional) integer index as input and returns True if the stopping condition has been met, and False otherwise.

find_stopping_index(metrics_dict)

Computes the index (active learning iteration) at which this StoppingCondition WOULD have stopped active learning if it were applied to the metrics for an already-trained BFBLearner object.

Parameters:

metrics_dict (dict) – A dictionary relating the names of ALMetric objects (or rather child classes of this class) to the objects themselves, extracted from a trained BFBLearner object.

Returns:

An integer representing the active learning round at which the StoppingCondition would have stopped active learning, if it had been implemented during training. If the condition would not have been met, returns -1.

Return type:

int

class bfbrain.AL_Metrics.ScoreNotDecreasing(metric_name, reduction='mean', patience=5)

Bases: StoppingCondition

A stopping condition based on when an uncertainty score (in particular BALD or variation ratios) is not decreasing over some data set (usually the pool of candidate points proposed by the classifier or the set of training points added as training data). Because mutual information, variation ratios, and predictive variance are all in theory metrics of epistemic uncertainty (or in the case of the second, at least highly sensitive to it), some measurement of these scores should be decreasing as more data is added. If it’s not, then the network probably reached close to the highest performance it’s capable of attaining.

metric_name

Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track

Type:

str

metric_func

A callable which takes a metric and an (optional) integer index as input and returns True if the stopping condition has been met, and False otherwise. In this case, metric_func checks to see if a specified uncertainty score on some set of points hasn’t achieved a new minimum over some specified number of rounds.

Type:

callable

reduction

Must be some reduction over the score that the metric specified by metric_name has evaluated. This is the specific quantity that the stopping condition monitors.’

Type:

{‘mean’, ‘min’, ‘max’, ‘std’}

patience

The number of rounds without achieving a new minimum for its monitored quantity that the stopping condition tolerates before halting active learning.

Type:

int, default=5

Parameters:

• metric_name (str) – Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track.

• reduction ({'mean', 'min', 'max', 'std'}) – Must be some reduction over the score that the metric specified by metric_name has evaluated. This is the specific quantity that the stopping condition monitors.’

• patience (int, default=5) – The number of rounds without achieving a new minimum for its monitored quantity that the stopping condition tolerates before halting active learning.

class bfbrain.AL_Metrics.AccuracyNotImproving(metric_name, patience=5)

Bases: StoppingCondition

A stopping condition that monitors either a ModelEvaluation or MCModelEvaluation metric and stops the active learning after some number of rounds have passed without achieving a new maximum accuracy.

metric_name

Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track

Type:

str

metric_func

A callable which takes a metric and an (optional) integer index as input and returns True if the stopping condition has been met, and False otherwise. In this case, metric_func checks to see if the accuracy entry for a ModelEvaluation or MCModelEvaluation metric hasn’t achieved a new maximum over some specified number of rounds.

Type:

callable

patience

The number of rounds without achieving a new maximum for its monitored quantity that the stopping condition tolerates before halting active learning.

Type:

int, default=5

Parameters:

• metric_name (str) – Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track.

• patience (int, default=5) – The number of rounds without achieving a new maximum for its monitored quantity that the stopping condition tolerates before halting active learning.

class bfbrain.AL_Metrics.FScoreNotImproving(metric_name, quant=1.0, patience=5)

Bases: StoppingCondition

A stopping condition that monitors a ValidationConfusionMatrix or ValidationFScore metric and stops the active learning after some number of rounds have passed without achieving a new maximum F score.

metric_name

Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track

Type:

str

metric_func

A callable which takes a metric and an (optional) integer index as input and returns True if the stopping condition has been met, and False otherwise. In this case, metric_func checks to see if the F score evaluated over some uncertainty quantile (see ValidationConfusionMatrix and ValidationFScore documentation for details) hasn’t achieved a new maximum in patience rounds.

Type:

callable

quant

The uncertainty quantile that the stopping condition should check. Default is 1.0, meaning the entire validation set is considered.

Type:

float, default=1.0

patience

The number of rounds without achieving a new maximum for its monitored quantity that the stopping condition tolerates before halting active learning.

Type:

int, default=5

Parameters:

• metric_name (str) – Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track

• quant (float, default=1.0) – The uncertainty quantile that the stopping condition should check. Default is 1.0, meaning the entire validation set is considered.

• patience (int, default=5) – The number of rounds without achieving a new maximum for its monitored quantity that the stopping condition tolerates before halting active learning.

class bfbrain.AL_Metrics.DeltaFNotDecreasing(metric_name, patience=5)

Bases: StoppingCondition

A stopping condition that monitors a PoolDeltaF or UnlabelledDeltaF metric and stops active learning once the classifier’s estimated change in F score has not decreased for a specified number of rounds.

metric_name

Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track.

Type:

str

metric_func

A callable which takes a metric and an (optional) integer index as input and returns True if the stopping condition has been met, and False otherwise. In this case, metric_func checks to see if the estimated change in F score has not achieved a new minimum in patience rounds.

Type:

callable

patience

The number of rounds without achieving a new maximum for its monitored quantity that the stopping condition tolerates before halting active learning.

Type:

int, default=5

Parameters:

• metric_name (str) – Denotes the name of the performance metric (that is, the ALMetric object’s name str) that the StoppingCondition should track

• patience (int, default=5) – The number of rounds without achieving a new maximum for its monitored quantity that the stopping condition tolerates before halting active learning.

bfbrain.BFB_Learner module

This module contains the core of BFBrain’s training capabilities, in particular the class BFBLearner, the object which contains the neural network classifier and methods to execute the active learning training loop.

bfbrain.BFB_Learner.create_seq_network(lam_len, n_layers, n_neurons, l, N)

Create a sequential Bayesian neural network approximated by concrete dropout.

Parameters:

• lam_len (int) – The number of quartic coefficients needed to uniquely specify a potential in the model.

• n_layers (int) – The number of hidden layers of neurons to include in the model. Generally recommended to be O(a few)

• n_neurons (int) – The number of neurons in each dense layer. Recommended to be O(100).

• l (float) – The prior length scale parameter of the neural network. Weights have a prior distribution of N(0, 1/l**2).

• N (int) – The number of entries in the training data. Needed to determine the appropriate loss regularization terms in concrete dropout.

Returns:

Returns a neural network

Return type:

tf.keras.Sequential

class bfbrain.BFB_Learner.AL_history

Bases: object

Holds cumulative information about the model performance at each epoch. An instance of this class is saved as part of the BFBLearner object.

losses

Represents the loss values at each training epoch of the neural network model, calculated over the training set.

Type:

list(float)

accuracy

Represents the binary accuracy at each training epoch of the neural network model, calculated over the training set.

Type:

list(float)

val_losses

Represents the loss values at each training epoch of the neural network model, calculated over the validation set. Only tracked if the validation performance is tracked during the call to model.fit during training, which the current BFBrain.BFBLearner.AL_loop method doesn’t do.

Type:

list(float)

val_accuracy

Represents the binary accuracy at each training epoch of the neural network model, calculated over the validation set. Only tracked if the validation performance is tracked during the call to model.fit during training, which the current BFBrain.BFBLearner.AL_loop method doesn’t do.

Type:

list(float)

Parameters:

None –

append_history(history)

Merges information from the latest active learning iteration, extracted from the output of the Tensorflow fit method, into the AL_history object.

Parameters:

history (Tensorflow history object) –

plot_history(filepath=None)

Plots the data stored in AL_history.

Parameters:

filepath (str, optional) – If specified, saves the resulting plots as .png images to the directory named in filepath. If not specified, the method just displays them on the console.

class bfbrain.BFB_Learner.BFBLearner(dm, model, data_train, data_val, metrics, history, learning_rate, hop_dist, rand_fraction, idx, l_constant)

Bases: object

Class which controls the active learning loop. Holds the model, the training and validation data, and some information about the training so far, and includes methods which perform the training loop and save the model for further training or exporting.

dm

The DataManager object that handles the generation and labelling of new training and validation data.

Type:

DataManager

model

The sequential neural network that classifies potentials as bounded-from-below.

Type:

tf.keras.Sequential

data_train

The data on which the neural network is trained. Is periodically augmented during the active learning loop.

Type:

np_data

data_val

A separate np_data object on which the neural network performance can be tested. If metrics doesn’t include any performance metrics on validation data, this should be None.

Type:

np_data or None

metrics

A list of objects which inherit from the abstract class ALMetric. These will represent the performance metrics that are tracked over each active learning iteration.

Type:

list(BFBrain.ALMetric)

history

An AL_history object which tracks the training loss and binary accuracy over each epoch.

Type:

AL_history

learning_rate

The learning rate used for the Adam optimizer during neural network training.

Type:

float

hop_dist

The distance scale used to generate new training points in the vicinity of existing bounded-from-below points, given as the hop_dist argument to DataManager’s generate_L function.

Type:

float

rand_fraction

The percentage of points generated by DataManager’s generate_L function that are randomly sampled instead of sampled in the vicinity of positively labelled points.

Type:

float

idx

The number of active learning rounds that the model has completed.

Type:

int

l_constant

The prior length scale for the neural network weights. The neural network is constructed so that the prior distribution on the weights is N(0, 1/l**2).

Type:

float

init_for_first_run()

The recommended constructor for initializing a BFBLearner object from scratch.

from_file()

The constructor for loading a saved BFBLearner object.

save_AL_state()

Used to save the BFBLearner object.

redefine_model()

Used to replace the neural network with a new one (for example adjusting weight priors, or changing the number of hidden neurons/layers). A use case would be generating a single BFBLearner object with a large labelled validation set, saving the object with an untrained neural network, and then loading the object in different contexts to experiment with different neural network architectures and hyperparameters.

set_l_constant()

Sets l_constant to a new value. Same use case as redefine_model.

add_metrics()

Adds new children of the ALMetric class to the BFBLearner object. Use this method rather than directly appending objects to the metrics attribute, since otherwise errors can occur, for example from adding a metric which requires a validation data set when the BFBLearner object had no previous validation data set.

plot_metrics()

Creates simple plots of all the metrics recorded in metrics. Useful to get a quick visual sense of the performance of the model after active learning finishes.

AL_loop()

The core active learning function. Executes the active learning loop to train the classifier. Highly customizable.

classmethod from_file(directory)

A constructor which loads an instance of BFBLearner from a folder.

Parameters:

directory (str) – Should denote a directory into which a previous BFBLearner object was saved.

classmethod init_for_first_run(dm, n_layers, n_neurons, metrics, nlams, nlams_val=-1, learning_rate=0.001, rand_fraction=0.0, hop_dist=None, l_constant=0.1, truth_label_fn=None, use_truth_for_train=False, val_label_kwargs=None, balance_val=False)

The recommended constructor when not loading a saved BFBLearner object.

Parameters:

• dm (DataManager) – The DataManager object that will handle the generation and processing of training and validation data.

• n_layers (int) – The number of layers of hidden layers in the neural network.

• n_neurons (int) – The number of neurons in each hidden layer.

• metrics (list(BFBrain.ALMetric)) – A list of objects which inherit from the abstract class ALMetric. These will represent the performance metrics that are tracked over each active learning iteration.

• nlams (int) – The number of sets of quartic potential coefficients to generate to produce the initial training data.

• nlams_val (int, optional) – The number of sets of quartic potential coefficients to generate to produce the validation data. If not specified, the value 100*nlams is used.

• learning_rate (float, default=0.001) – The learning rate used for the Adam optimizer during neural network training.

• rand_fraction (float, default=0.) – The percentage of points generated by DataManager’s generate_L function that are randomly sampled instead of sampled in the vicinity of positively labelled points.

• hop_dist (float, optional) – The distance scale used to generate new training points in the vicinity of existing bounded-from-below points, given as the hop_dist argument to DataManager’s generate_L function. If not specified, the value is estimated using BFBrain.Active_Learning._get_hop_dist

• l_constant (float, default=0.1) – The prior length scale for the neural network weights. The neural network is constructed so that the prior distribution on the weights is N(0, 1/l**2).

• truth_label_fn (callable, optional) – Must take a 1-D NumPy array representing a single set of quartic coefficients and return a Boolean True if the potential they describe is bounded from below, False otherwise. If this argument is specified, the method will use this callable to label the validation data set instead of DataManager’s oracle, and if use_truth_for_train is True, then it will also use this function to label the initial training data. This is used in specific instances when a fast symbolic expression for the bounded-from-below constraints is known, and the performance of the classifier training loop can be evaluated in the absence of noise due to the approximate labeller. Obviously the use case of the classifier is for potentials where such a symbolic expression is NOT known, so the real-world model building usefulness of this option is limited.

• use_truth_for_train (bool, default=False) – If True, use truth_label_fn to label the initial training data instead of the DataManager’s oracle function. Has no effect unless truth_label_fn is specified.

• val_label_kwargs (dict, optional) – An alternate set of keyword arguments for the oracle that can be specified when labelling the validation data instead of using the settings in the DataManager object. Useful if, for example, a less expensive oracle can be used for the validation labelling than for labelling the training data, or if trying to gauge the effect of using a noisier oracle during training while still verifying performance with reliably labelled validation data.

• balance_val (bool, default=False) – If True, balance the validation data set between positive and negative examples using DataManager’s balance_array method, making binary accuracy of the classifier more informative at the cost of rendering the generating distribution for the validation data considerably more opaque. Generally recommended to leave False, since other performance metrics, such as F score, can help gauge the classifier performance without needing to rebalance the validation data.

set_l_constant(l_constant)

Sets the weight prior length scale l to a new value and updates the network to reflect it.

Parameters:

l_constant (float) – The new value for the class attribute l_constant, the weight prior length scale.

save_AL_state(directory)

A method for saving all the relevant states for the BFBLearner object in a directory for later retrieval.

Parameters:

directory (str) – A string denoting a directory to which the data needed to reconstruct the BFBLearner object will be saved.

redefine_model(n_layers, n_neurons, l=None, learning_rate=0.001)

A method for redefining the parameters of the neural network. Should ONLY be called if the neural network is entirely untrained.

Parameters:

• n_layers (int) – The number of hidden layers of neurons to include in the model. Generally recommended to be O(a few).

• n_neurons (int) – The number of neurons in each hidden layer. Recommended to be O(100).

• l (float, optional) – The prior length scale parameter of the neural network. Weights have a prior distribution of N(0, 1/l**2). If l isn’t specified, the BFBLearner object’s current value for the parameter is used.

• learning_rate (float, default=0.001) – The learning rate of the Adam optimizer for neural network training.

add_metrics(metrics, nlams_val=100000)

add new metrics to the BFBLearner instance after initialization. Appends any metric or list of metrics specified to self.metrics. If an added metric involves measuring the classifier performance on a validation set, and the ActiveLearning instance doesn’t have a validation set yet, this method will generate one.

Parameters:

• metrics (ALMetric or list of ALMetrics) – A metric or list of objects which inherit from the BFBrain.ALMetric abstract class.

• nlams_val (int, default=100000) – The number of points to be created and labelled to produce a validation set, if a new validation set needs to be generated.

plot_metrics(filepath=None)

A method to plot all metrics being recorded in the BFBLearner’s metrics object.

Parameter

filepathstr, optional

If specified, the method will save the plots to the folder specified in filepath (provided that folder exists). If not specified, simply prints the plots to the console.

get_calibration_uncertainties(score_fn, nlams=1000000, n_trials=None, batch_size=200000)

A function which gets a sense of the range of uncertainty values (and what constitutes a highly uncertain point). It does this by creating a large sample of unlabelled data points by uniformly sampling the unit hypersphere, and getting the outputs of a specified uncertainty score over the points in the distribution which the model predicts to be positive.

Parameters:

• score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Either a string which denotes which of the implemented strategies for scoring points to add to the training set to employ, or a callable representing a custom function to perform this role. Any custom functions must have a signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether n_trials is specified or not. The string inputs correspond to scoring based on, in order: Mutual information, Shannon entropy, variation ratios, a random score, query by dropout committee (QBDC), and standard deviation of the neural network predictions.

• nlams (int, default=1000000) – The number of sets of quartic potential coefficients that the method should generate

• n_trials (int, optional) – Many of the implemented score_fn methods (‘BALD’, ‘MaxEntropy’, ‘variation_ratios’, ‘QBDC’, and ‘predictive_variance’) have an optional argument controlling the number of forward passes through the network to compute their values. This parameter allows this value to be specified for score_fn’s evaluations.

• batch_size (int, default=200000) – The size of batches of the elements of lams that are transferred to the GPU simultaneously. If OOM errors are encountered, it is recommended to reduce batch_size.

AL_loop(filepath='saved_AL', K_batch_size=500, K_batch_num=10, K_factor=100, score_fn='BALD', nstop=20, full_train_interval=1, epoch_patience=100, epoch_limit=20000, batch_size=200000, val_batch_size=-1, save_interval=5, stopping_cond=None, prob_score_fn=None, truth_label_fn=None, score_ntrials=None, prob_score_ntrials=None, reinitialize_weights=True, rebalance_train_data=False, L_probability_weighting=False, plot_metrics=False, verbose=False)

The core function for the active learning loop. Performs active learning for a specified number of rounds with a customizable query strategy and a number of options exposed to the user. BFBrain analysis principally consist of execution(s) of this method to train the BFBLearner’s classifier.

Parameters:

• filepath (str, default='saved_AL') – A string describing the name of a directory (which will be created, if it doesn’t exist) into which the Active_Learning object will be saved after training.

• K_batch_size (int, default=500) – Additional training data, called K, is generated in batches of K_batch_size entries. After each cycle of training, K_batch_num batches, each with K_batch_size entries, are generated and added to the training data.

• K_batch_num (int, default 10) – The number of batches, of batch_size K_batch_size, of additional training data that is generated after each cycle of training.

• K_factor (int, default 100) – To generate each batch of K_batch_size entries, K_factor*K_batch_size candidate points are generated and the top K_batch_size points are selected to be added to K.

• score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Either a string which denotes which of the implemented strategies for scoring points to add to the training set to employ, or a callable representing a custom function to perform this role. Any custom functions must have a signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether score_ntrials is specified or not. The string inputs correspond to scoring based on, in order: Mutual information, Shannon entropy, variation ratios, a random score, expected gradient length, query by dropout committee (QBDC), and variance of the neural network predictions.

• nstop (int, default=20) – The number of total cycles (generating new training data, fitting the neural network to the training data, and recording metrics) to perform.

• full_train_interval (int, default=1) – The active learning function will train over the entire training data set every full_train_interval cycles. Otherwise, it will only train the network on the most recently generated set of new training data, for a much smaller number of epochs. Recommended setting is 1, so that dropout approximation to a Bayesian neural network holds rigorously.

• epoch_patience (int, default=100) – A parameter for the training in individual cycles. The neural network will stop training early in each cycle if the network’s performance on the validation data has not improved for epoch_patience epochs.

• epoch_limit (int, default=20000) – The maximum number of epochs for the neural network to train each cycle.

• batch_size (int, default=200000) – The size of batches of training data that are transferred to the GPU simultaneously. If OOM errors are encountered, it is recommended to reduce batch_size. Some discussion of batch size is in order. We find that for greatest training stability, it is best to have a batch_size larger than the maximum number of training examples that will be included in an active learning iteration– so for starting at 1000 initial points, with 5000 points added at each iteration and 20 iterations executed, a batch_size > 101000 may be recommended. This will maximize training stability. However, if instead more training examples must be considered than can realistically fit into GPU memory, we recommend batch_size parameters of approximately the number of points added during each active learning iteration, to avoid different active learning iterations suddenly exhibiting radically different performance as the number of batches abruptly changes for the first time late into active learning. Experimentation regarding small batch size for different potentials is recommended.

• val_batch_size (int, optional) – The size of batches of validation data that are transferred to the GPU simultaneously. May be useful if small batches are used for training, but much larger batches can be accomodated during validation. If not specified, batch_size will be used to batch the validation data.

• save_interval (int, default=5) – Every save_interval active learning iterations, the function will save the BFBLearner object to the directory specified in filepath.

• stopping_cond (BFBrain.StoppingCondition object or subclass) – This object allows a user to specify conditions under which active learning should terminate before nstop iterations have been performed. See the BFBrain.StoppingCondition documentation for more details.

• prob_score_fn ({'BALD', 'MaxEntropy', 'variation_ratios', 'random', 'QBDC', 'predictive_variance'} or callable) – Either a string which denotes which of the implemented strategies for scoring points to add to the training set, or a callable representing a custom function to perform this role. Any custom functions must have a signature (tf.keras.model, tf.tensor(tf.float32, tf.float32))-> tf.tensor(tf.float32) or (tf.keras.model, tf.tensor(tf.float32, tf.float32), int)-> tf.tensor(tf.float32), depending on whether score_ntrials is specified or not. Unlike score_fn, this function is employed when weighting points to sample around to generate new training data, so that an algorithm might preferentially sample around points with high mutual information (with prob_score_fn = ‘BALD’), but decide which points to add to the training set based on Shannon entropy (score_fn = ‘MaxEntropy’), for example. Has no effect if L_probability_weighting is False.

• truth_label_fn (callable, optional) – If a callable, must take a 1-D NumPy array representing a single set of quartic coefficients and return a Boolean True if the potential they describe is bounded from below, False otherwise. If specified, training data will be labelled using this callable instead of the oracle in the DataManager object.

• score_ntrials (int, optional) – Many of the implemented score_fn methods (‘BALD’, ‘MaxEntropy’, ‘variation_ratios’, ‘QBDC’, and ‘predictive_variance’) have an optional argument controlling the number of forward passes through the network to compute their values. This parameter allows this value to be specified for score_fn’s evaluations.

• prob_score_ntrials (int, optional) – Same as score_ntrials, but for prob_score_fn. Does not have any effect if prob_score_fn is not specified.

• reinitialize_weights (bool, default=True) – If True, randomize the neural network weights before each new round of active learning (unless the algorithm is not training on the full training data set– see full_train_interval). The recommended setting is True, to prevent potential overfitting on the points sampled earlier and maintain the dropout approximation of the Bayesian neural network rigorously.

• rebalance_train_data (bool, default=False) – If True, the training set will be rebalanced every iteration by adding new bounded from below points to correct an overabundance of points which are not bounded from below.

• L_probability_weighting (bool, default=False) – If True, the selection of existing bounded-from-below training points to sample around for generating new points will be weighted based on prob_score_fn.

• plot_metrics (bool, default=False) – If True, all metrics will be plotted in the command line as well as saved to the output filepath when training completes.

• verbose (bool, default=False) – If True, print out information about the progress and performance of the active learning loop throughout the run to the console. Otherwise data on the neural network performance after each iteration is still saved to a file ‘output.txt’ in the filepath directory.

bfbrain.Custom_Layers module

This module contains the various neural network layers used by BFBrain.

class bfbrain.Custom_Layers.HypersphereProjectionLayer(*args, **kwargs)

Bases: Layer

A custom neural network preprocessing layer which projects any input quartic coefficients onto the unit hypersphere.

build(input_shape)

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

classmethod from_config(config)

Creates a layer from its config.

This method is the reverse of get_config, capable of instantiating the same layer from the config dictionary. It does not handle layer connectivity (handled by Network), nor weights (handled by set_weights).

Parameters:

config – A Python dictionary, typically the output of get_config.

Returns:

A layer instance.

call(inputs)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

bfbrain.Custom_Layers.get_weight_regularizer(N, l=0.01, tau=0.1)

Determines the weight decay constant which should be applied in the loss function with a given precision, prior length scale, and number of training data points.

Parameters:

• N (int) – The number of data points in the training data.

• l (float, default=1e-2) –

• tau (float, defulat = 0.1) – neural network precision. For classification networks this is just set to 1.

Return type:

float

bfbrain.Custom_Layers.get_dropout_regularizer(N, tau=0.1, cross_entropy_loss=False)

Controls the regularization term associated with the entropy of the cells’ dropout probabilities.

Parameters:

• N (int) – The number of data points in the training data.

• tau (float, defulat = 0.1) – neural network precision. For classification networks this is just set to 1.

• cross_entropy_loss (bool, default=False) – Should be True if the loss function is cross entropy (so the neural network is a classifier), and False otherwise.

Return type:

float

class bfbrain.Custom_Layers.ConcreteDenseDropout(*args, **kwargs)

Bases: Dense

Code for the implementation of concrete dropout. Based heavily on https://github.com/aurelio-amerio/ConcreteDropout, a Tensorflow 2.0 implementation of the concrete dropout algorithm described in arXiv:1705.07832. Modified from that implementation in order to save the model more easily at the expense of some flexibility. IMPORTANT: these layers perform dropout BEFORE the wrapped operation.

build(input_shape=None)

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

spatial_concrete_dropout(x, p)

call(inputs, training=None)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

bfbrain.Data_Manager module

A module containing the DataManager class, which handles the generation and labelling of training and validation data for the BFBLearner class.

class bfbrain.Data_Manager.labeller_wrapper(func, phi_len, lam_len, rng, polar, label_fn=None, label_check=None, label_kwargs={'niter': 250})

Bases: object

Wrapper for the labelling function which serves as the active learning oracle.

func

A numeric function for the potential. This is a numeric function generated by the class DataManager in its init method. It will take numeric arrays (of a format depending on the DataManager class) representing a scalar vev and a set of quartic potential coefficients and return the numerical value of the quartic part of the potential function and its gradient with respect to the vev.

Type:

callable

phi_len

The number of real parameters necessary to uniquely specify a vev in the model.

Type:

int

lam_len

The number of independent real quartic coefficients in the model.

Type:

int

rng

The random number generator which governs any random processes that the oracle may use.

Type:

np.random.Generator

polar

If True, then the analysis of the potential will be conducted with a polar coordinate parameterization of the vev parameters. If False, then Cartesian coordinates will be used, albeit with the vev parameters restricted to a phi_len-dimensional unit hypersphere.

Type:

bool

label_fn

A function that takes a 2-D NumPy array of quartic coefficients and returns a list of Boolean labels for them. This is for implementing customized oracle functions. Must have the signature (func: Callable, phi_len: int, polar: bool, rng: NumpyGenerator, lam: np.array(np.float, np.float), **kwargs) -> np.array(bool) If this argument is not specified, the default oracle BFBrain.Jax_Oracle.label_func is used.

Type:

callable, optional.

label_check

A function that can be used to test the reliability of a custom oracle given by label_fn, or if label_fn is None, the default oracle BFBrain.Jax_Oracle.label_func. Must have the same signature as label_fn, up to additional keyword arguments. If this argument is not specified, the a tester for the default oracle is used: BFBrain.Jax_Oracle.test_labeller

Type:

callable, optional

\*\*label_kwargs

A dictionary of additional keyword arguments needed for the labelling function label_func. The default values are applicable for the default oracle function, BFBrain.Jax_Oracle.label_func

Type:

dict, default=dict(niter = 250)

do_labelling(lam, label_kwargs=None)

Performs labelling using the class’s oracle function.

Parameters:

• lam (np.array(np.float32, np.float32)) – A 2-D NumPy array of sets of quartic potential coefficients.

• label_kwargs (dict, optional) – An optional alternative set of oracle keyword arguments. If not specified, the class instance’s label_kwargs attribute is used.

Returns:

A 1-D NumPy array of labels for lam, for which points that are bounded from below are labelled “True” and points which are not are labelled “False”.

Return type:

np.array(bool)

check_labeller(lam, **tester_kwargs)

Tests the reliability of the labelling– calls self.label_check. Depending on the methodology of label_func, this function may or may not be useful. For example, a rigorous computation of boundedness-from-below based on resultants would not require any consistency or reliability checks.

Parameters:

• lam (np.array(np.float32, np.float32)) – A 2-D NumPy array of sets of quartic potential coefficients.

• tester_kwargs (dict) – A set of keyword arguments for self.label_check.

Returns:

Will return what self.label_check returns.

Return type:

Any

class bfbrain.Data_Manager.np_data(pos, neg)

Bases: object

Holds labelled sets of quartic coefficients in CPU memory in a format that’s easy to save, load, and manipulate.

pos

A 2-D NumPy array of sets of quartic coefficients in the potential, which the labeller has determined are bounded-from-below.

Type:

np.array(np.float32, np.float32)

neg

A 2-D NumPy array of sets of quartic coefficients in the potential, which the labeller has determined are NOT bounded-from-below.

Type:

np.array(np.float32, np.float32)

classmethod from_file(path)

A constructor for loading an np_data object from a .npz file (see NumPy documentation), likely created in a previous BFBrain analysis.

Parameters:

path (str) – A string with a file name. ‘.npz’ is appended to the end of the string, and should not be included in path.

Return type:

np_data

save_data(path)

Saves the data object to the filepath specified as an npz object (see NumPy documentation)

Parameters:

path (str) – A string with a file name. If .npz is not at the end of the string, it is appended to it.

append_data(new_data)

Given another np_data object, appends its data to this object in place.

Parameters:

new_data (np_data) –

n_elements()

Computes the total number of sets of quartic coefficients in the object (both bounded-from-below and not bounded-from-below)

Returns:

The total number of sets of quartic coefficients in the np_data object.

Return type:

int

class bfbrain.Data_Manager.DataManager(phi_len, lam_len, rng, polar, sym_expr, sym_grad_expr, phisym_var, lamsym, lambdify_mode='jax', label_fn=None, label_check=None, **label_kwargs)

Bases: object

A class containing methods which process and generate data. Note that this class contains all the random number generation that’s not specifically associated with the neural network and its optimizer. Generally one should use the ‘from_seed’ or ‘from_file’ constructor rather than constructing from the base initialization method.

phi_len

The number of independent real parameters needed to uniquely specify a vev in the model.

Type:

int

lam_len

The number of independent real quartic potential coefficients in the model.

Type:

int

rng

A list of NumPy random number generators which control all the random generation related to the generation and labelling of data. In total there are 6 random number generators, each differently seeded using NumPy’s SeedSequence.spawn method. Each random number generator is used only for one specific task: Generating training data, generating validation data, generating random points in the vicinity of other points (two rng’s are used here, one for rotation direction and the other for rotation angle), doing random number generation associated with labelling, and shuffling data for training.

Type:

list of numpy.random.Generator

polar

If true, the potential is analyzed with the vev coordinates converted to a polar form. If false, they are analyzed in their Cartesian form.

Type:

bool

sym_expr

Represents the potential function in a form that is both picklable and can easily be used to generate the gradient symbolically.

Type:

SymPy expression

sym_grad_expr

Represents the gradient of the potential function in a form that is both picklable and can easily be used to generate the gradient symbolically.

Type:

SymPy expression

phisym_var

The symbols representing the quartic potential coefficients in sym_expr.

Type:

sympy.Array

lamsym

The symbols representing the quartic potential coefficients in sym_expr.

Type:

sympy.Array

lambdify_mode

Passed directly as the argument “modules” in sympy.lambdify, the function used to generate numerical functions from the symbolic expression for the scalar potential. Default value is ‘jax’, consistent with the default oracle function, BFBrain.Jax_Oracle.label_func

Type:

{‘jax’, ‘numpy’, ‘scipy’, ‘math’, ‘mpmath’, ‘numexpr’, ‘sympy’, ‘tensorflow’}

label_fn

A function that takes a 2-D NumPy array of quartic coefficients and returns a list of Boolean labels for them. This is for implementing customized oracle functions. Must have the signature (func: Callable, phi_len: int, polar: bool, rng: NumpyGenerator, lam: np.array(np.float, np.float), **kwargs) -> np.array(bool) If this argument is not specified, the default oracle BFBrain.Jax_Oracle.label_func is used.

Type:

callable, optional

label_check

A function that can be used to test the reliability of a custom oracle given by label_fn, or if label_fn is None, the default oracle BFBrain.Jax_Oracle.label_func. Must have the same signature as label_fn, up to additional keyword arguments. If this argument is not specified, the a tester for the default oracle is used: BFBrain.Jax_Oracle.test_labeller

Type:

callable, optional

\*\*label_kwargs

A dictionary of additional keyword arguments needed for the labelling function label_func.

Type:

dict, optional

classmethod from_func(sym_func, phi_len, lam_len, seed=None, polar=False, lambdify_mode='jax', label_fn=None, label_check=None, **label_kwargs)

Preferred constructor for initializing DataManager.

Parameters:

• sym_func (SymPy function.) – A SymPy function that expresses the quartic part of the potential. Must have the signature (sympy.Array, sympy.Array) -> sympy.Expr, where the first sympy.Array object corresponds to the vev configuration and the second corresponds to the quartic coefficients in the potential.

• phi_len (int) – The number of real parameters needed to uniquely specify the vev in the model.

• lam_len (int) – The number of independent real quartic coupling coefficients in the model’s potential function.

• seed (int, optional) – A random number seed. Used to spawn a sequence of random generators with SeedSequence.

• polar (bool, default=False) – If true, the potential is analyzed with the vev coordinates converted to a polar form. If false, they are analyzed in their Cartesian form.

• lambdify_mode ({'jax', 'numpy', 'scipy', 'math', 'mpmath', 'numexpr', 'sympy', 'tensorflow'}) – The “module” input to sympy.lambdify, used to extract numerical expressions from the symbolic SymPy function. See SymPy documentation for details.

• label_fn (callable, optional) – A function that takes a 2-D NumPy array of quartic coefficients and returns a list of Boolean labels for them. This is for implementing customized oracle functions. Must have the signature (func: Callable, phi_len: int, polar: bool, rng: numpy.random.Generator, lam: np.array(np.float, np.float), **kwargs) -> np.array(bool) If this argument is not specified, the default oracle BFBrain.Jax_Oracle.label_func is used.

• label_check (callable, optional) – A function that can be used to test the reliability of a custom oracle given by label_fn, or if label_fn is None, the default oracle BFBrain.Jax_Oracle.label_func. Must have the same signature as label_fn, up to additional keyword arguments. If this argument is not specified, the a tester for the default oracle is used: BFBrain.Jax_Oracle.test_labeller

• **label_kwargs (dict, optional) – A dictionary of additional keyword arguments needed for the labelling function label_func.

create_random_lambdas(nlams, validation=False)

Create a list of random sets of quartic potential coefficients (but don’t label them yet). Use independent uncorrelated rng’s for the generation of a validation and training set. Notice that these lambdas are Cartesian coordinates that uniformly sample the unit hypersphere.

Parameters:

• nlams (int) – The number of sets of quartic coefficients to generate randomly.

• validation (bool, default=False) – A flag denoting which random number generator to use, ensuring independently-generated validation and training sets. If True, use the random number generator for the validation set, while if False, use the random number generator for the training set.

Returns:

A 2-D NumPy array representing a list of sets of quartic coefficients for the potential.

Return type:

np.array(np.float32, np.float32)

check_labeller(nlams, **tester_kwargs)

A wrapper for calling the labeller’s check_labeller function. Generates sample quartic coefficients randomly before running labeller.check_labeller on them.

Parameters:

• nlams (int) – The number of sets of quartic coefficients to randomly generate for testing the labeller function’s consistency.

• tester_kwargs (dict) – Additional keyword arguments required by the check_labeller function. If the default oracle and tester are used, possible keyword arguments are niter_step, count_success, max_iter, verbose. See BFBrain.Jax_Oracle.test_labeller for details.

Returns:

• Same type as labeller.check_labeller, which may be a

• user-written function. If the default oracle and tester are

• used, this function will be BFBrain.Jax_Oracle.test_labeller.

checklam_all(lams, truth_label_fn=None, label_kwargs=None)

Labels sets of quartic potential coefficients with True (for bounded from below) or False (not bounded from below).

Parameters:

• lams (np.array(np.float32, np.float32)) – A 2-D NumPy array of quartic coefficients of the potential. Each entry along the 0 axis corresponds to a single set of quartic potential coefficients specifying a potential function.

• truth_label_fn (callable, optional) – Must take a 1-D NumPy array representing a single set of quartic coefficients and return a Boolean True if the potential they describe is bounded from below, False otherwise. If this argument is specified, the method will use this callable to label lams instead of the labeller class. This is used in specific instances when a fast symbolic expression for the bounded-from-below constraints is known, and the performance of the classifier training loop can be evaluated in the absence of noise due to the approximate labeller. Obviously the use case of the classifier is for potentials where such a symbolic expression is NOT known, so the real-world model building usefulness of this option is limited.

• label_kwargs (dict, optional) – If these are specified, the oracle will use the keyword arguments given here instead of the keyword arguments specified in the DataManager constructor.

Returns:

A Boolean NumPy array of labels for each set of coefficients in lams.

Return type:

np.array(bool)

create_data(lams, truth_label_fn=None, label_kwargs=None)

Given an unlabelled 2-D NumPy array of sets of quartic coefficients, label them and return an np_data object.

Parameters:

• lams (np.array(np.float32, np.float32)) – A 2-D NumPy array of quartic coefficients of the potential. Each entry along the 0 axis corresponds to a single set of quartic potential coefficients specifying a potential function.

• truth_label_fn (callable, optional) – Must take a 1-D NumPy array representing a single set of quartic coefficients and return a Boolean True if the potential they describe is bounded from below, False otherwise. If this argument is specified, the method will use this callable to label lams instead of the labeller class. This is used in specific instances when a fast symbolic expression for the bounded-from-below constraints is known, and the performance of the classifier training loop can be evaluated in the absence of noise due to the approximate oracle. Obviously the use case of the classifier is for potentials where such a symbolic expression is NOT known, so the real-world model building usefulness of this option is limited.

• label_kwargs (dict, optional) – If these are specified, the oracle will use the keyword arguments given here instead of the keyword arguments specified in the DataManager constructor.

Returns:

An np_data object representing the labelled contents of the input array lams.

Return type:

np_data

create_random_data(nlams, validation=False, truth_label_fn=None, label_kwargs=None)

Creates a random sample of Cartesian lambda coefficients and labels them, then storing the results in an np_data object.

Parameters:

• nlams (int) – The number of sets of quartic potential coefficients to generate.

• validation (bool, default=False) – If True, use the validation random number generator to generate the random coefficients. If False, use the training random number generator.

• truth_label_fn (callable, optional) – Must take a 1-D NumPy array representing a single set of quartic coefficients and return a Boolean True if the potential they describe is bounded from below, False otherwise. If this argument is specified, the method will use this callable to label lams instead of the labeller class. This is used in specific instances when a fast symbolic expression for the bounded-from-below constraints is known, and the performance of the classifier training loop can be evaluated in the absence of noise due to the approximate labeller. Obviously the use case of the classifier is for potentials where such a symbolic expression is NOT known, so the real-world model building usefulness of this option is limited.

• label_kwargs (dict, optional) – If these are specified, the oracle will use the keyword arguments given here instead of the keyword arguments specified in the DataManager constructor.

Returns:

An np_data object that represents the labelled sets of quartic coefficients that was randomly generated.

Return type:

np_data

check_accuracy_with_better_labeller(in_data, label_kwargs)

A method for evaluating the accuracy of a labeller which is capable of mislabelling some False points as True, like the default oracle, which is based on global minimization of the quartic part of the potential.

Parameters:

• in_data (np_data) – An np_data object, labelled with an oracle that can mislabel some False points as True (but not the reverse).

• label_kwargs (dict) – A dictionary which specifies the keyword arguments for the oracle, which must be selected to yield significantly more accurate labels than the ones specified by in_data.

Returns:

The precision (fraction of positively labelled points that are true positives) of the oracle which originally labelled in_data.

Return type:

float

balance_array(data)

Given an np_data object that has more negative (not bounded-from-below) points than positives (bounded-from-below), rebalance data to include new positive points generated by leveraging the convexity of the space of bounded-from-below points. If there are as many or more positive points as negative points in the np_data object, leaves the np_data object unmodified.

Parameters:

data (np_data) – The np_data object that has many more negatively-labelled points (that is, points which are not bounded-from-below) than positives.

create_dataset(data, validation=False)

Given an np_data object, creates a Tensorflow dataset object for training.

Paramaters

data : np_data

validationbool, default=False

If True, don’t shuffle the dataset. Useful for keeping track of agreement on the validation set for the model after successive active learning rounds.

returns:

This dataset is NOT batched, but is randomly shuffled on the CPU. To use for training, it is necessary to batch the dataset object with tf.data.Dataset’s batch method.

rtype:

tf.data.Dataset

generate_L(nL, lams, hop_dist, probs=None, rand_fraction=0.0)

Randomly generate a sample of new points in the vicinity of some existing points. Given some set of existing points, samples new points by making random hops of an angle given by a draw from a normal distribution in a random direction along the unit hypersphere in quartic coefficient space.

Parameters:

• nL (int) – The number of new sets of quartic coefficients to generate.

• lams (np.array(np.float32, np.float32)) – A 2-D NumPy array representing sets of quartic potential coefficients. New points will be sampled in the vicinity of these.

• hop_dist (float) – The distance scale for sampling around the coefficients in lams. Newly-generated points are taken from input points by randomly rotating points in lams by an angle taken from a normal distribution with standard deviation hop_dist.

• probs (np.array(np.float32), optional) – A 1-D NumPy array, should be an array of nonnegative floats which sum to 1, representing the probability of the function selecting each index of lams to generate new points around. If not specified, a uniform selection probability for all points in lams is used.

• rand_fraction (float, optional) – Must be a non-negative float between 0 and 1. If specified, then the method will sample that fraction of its points as uniformly distributed draws from the surface of the unit hypersphere in quartic coefficient space, instead of sampling in the vicinity of points in lams. If not specified, all generated points will be sampled in the vicinity of points in lams.

Returns:

A 2-D NumPy array representing a list of sets of quartic coefficients for the potential.

Return type:

np.array(np.float32, np.float32)

bfbrain.False_Proximity_Test module

These are a series of functions used to score models based on how distant their false positives and false negatives over some data set are from the model’s decision boundary. The only function used externally is combined_false_score, which discusses how the scoring is done. All other functions in this file are only used internally by combined_false_score and the functions that it calls.

bfbrain.False_Proximity_Test.combined_false_score(model, ds, dist=<tf.Tensor: shape=(), dtype=float32, numpy=0.05>)

A method to evaluate how “wrong” the model’s predictions of the validation set actually are, based on how far false positives and false negatives are from the decision boundary. Returns two sets of information for the false positives and the false negatives. For false positives (negatives), uses _find_accurate_points to find nearby sets of coefficients that are correctly classified as negative (positive). The angular distance between these new points and the corresponding false positive (negative) points is then computed in radians. The function returns the mean, standard deviation, and maximum of these distances for both false positives and negatives, as well as the number of points for which the angular distance exceeds a specified angle in radians.

Parameters:

• model (tf.keras.Model) –

• ds (tf.data.Dataset) – A labelled Tensorflow dataset of sets of quartic potential coefficients.

• dist (tf.float32) – A maximum angular distance in radians between an incorrectly classified point and the classifier’s decision boundary that the user deems acceptable. For small values of dist, this corresponds to the maximum difference in a (normalized) quartic potential coefficient between an incorrectly classified point and a correctly classified one.

Returns:

Each list contains the mean, standard deviation, and maximum of the angular distance between incorrectly classified points in ds and correctly classified ones generated with _find_accurate_points. The final element of each list gives the number of incorrectly classified points that are greater than dist radians away from a correctly classified one.

Return type:

tuple of lists of floats.

bfbrain.False_Proximity_Test.get_false_pos_and_neg_tf(model, ds)

A function which extracts all sets of quartic coefficients in a Tensorflow dataset that the neural network classifies incorrectly, either false positives (points it incorrectly classifies as bounded-from-below) or false negatives (points it incorrectly classifies as NOT bounded-from-below).

Parameters:

• model (tf.keras.Model) –

• ds (tf.data.Dataset) – A Tensorflow dataset representing labelled sets of quartic potential coefficients.

Returns:

Two 2-D tensors representing the sets of false positive and false negative quartic coefficients, respectively.

Return type:

tuple of tf.Tensors

bfbrain.Hypersphere_Formulas module

This module contains code for some basic manipulations to translate between n-dimensional polar and Cartesian coordinates.

bfbrain.Hypersphere_Formulas.convert_from_polar(v)

Converts an array of inputs in the range [0,pi) into an array of inputs in a Cartesian form

Parameters:

v (np.array(np.float32, np.float32)) – A 2-D NumPy array of points in polar coordinates

Returns:

A 2-D NumPy array of points in Cartesian coordinates on the surface of the unit hypersphere.

Return type:

np.array(np.float32, np.float32)

bfbrain.Hypersphere_Formulas.convert_to_polar(v)

Converts an array of inputs in Cartesian form into a lower-dimensional angular parameterization.

Parameters:

v (np.array(np.float32, np.float32)) – A 2-D NumPy array of points in Cartesian coordinates

Returns:

A 2-D NumPy array of points in polar coordinates.

Return type:

np.array(np.float32, np.float32)

bfbrain.Hypersphere_Formulas.jax_convert_to_polar(v)

Converts an array of inputs in Cartesian form into a lower-dimensional angular parameterization.

Parameters:

v (jnp.array(jnp.float32)) – A 1-D Jax NumPy array representing a point in Cartesian coordinates.

Returns:

A 2-D Jax NumPy array of points in polar coordinates.

Return type:

jnp.array(jnp.float32, jnp.float32)

bfbrain.Hypersphere_Formulas.jax_convert_from_polar(v)

Converts an array of inputs in the range [0,pi) into an array of inputs in a Cartesian form

Parameters:

v (jnp.array(jnp.float32)) – A 1-D Jax NumPy array represenging a point in polar coordinates.

Returns:

A 2-D Jax NumPy array of points in Cartesian coordinates on the surface of the unit hypersphere.

Return type:

jnp.array(jnp.float32, jnp.float32)

bfbrain.Hypersphere_Formulas.rand_nsphere(n_points, n_dims, rng)

Randomly generates points sampled uniformly from the surface of a unit hypersphere of specified dimension.

Parameters:

• n_points (int) – The number of points the method should generate.

• n_dims (int) – The dimensionality of the hypersphere that the method should sample on the surface of.

• rng (np.random.Generator) –

Returns:

A 2-D NumPy array representing sets of points uniformly sampled from the surface of the n_dims-dimensional unit hypersphere.

Return type:

np.array(np.float32, np.float32)

bfbrain.Hypersphere_Formulas.cumprod_sym(x, j)

Computes the cumulative product of some subset of elements of a 1-D SymPy array.

Parameters:

• x (sympy.Array) – A 1-D SymPy array of symbols.

• j (int) – The cumulative product will be computed by taking the product of the 0th through jth element of the array.

Return type:

sympy.symbol

bfbrain.Hypersphere_Formulas.convert_from_polar_sym(v)

Converts symbolic polar coordinates into symbolic Cartesian coordinates.

Parameters:

v (sympy.Array) – A 1-D SymPy array of symbols representing a set of polar coordinates.

Return type:

sympy.Array

bfbrain.Jax_Oracle module

The code in this module contains BFBrain’s default algorithm for labelling if a point is bounded from below. It is designed to work with the methods in Active_Learning.py. The fundamental strategy of this labelling method is to take a quartic part of the potential function with fixed quartic coefficients, and attempt many consecutive local optimizations with respect to the scalar vev with random initial starting points. If it can find a point where the potential is negative, it will label the set of quartic coefficients as not bounded from below, while if after some user-specified number of local minimization iterations the found minima are always positive, the set will be labelled as bounded from below.

bfbrain.Jax_Oracle.take_step(x0, key, polar)

Return a tuple of phi, key (in that order) after randomly generating phi as a vev point on the unit hypersphere.

Parameters:

• x0 (jnp.array(jnp.float32)) – A Jax NumPy array that tells the function what shape the vev input arrays should be.

• key (Jax PRNGGKey) – A key object used to generate random numbers in Jax. Will be consumed over the running of the function and a new key will be returned.

• polar (bool) – A flag denoting whether the labeller should use Cartesian coordinates for the vev (False) or convert them into polar coordinates on the unit hypersphere (True).

Returns:

• rand_phi (jnp.array(jnp.float32).) – A 1-D Jax Numpy array representing a single point in vev space.

• new_key (Jax PRNGKey) – A key object used to generate random numbers in Jax. Supplied to replace the consumed key.

bfbrain.Jax_Oracle.run_one_step(key, minimizer, stepper, lam, polar)

Takes a single local minimization step. This consists of randomly generating a starting point, running the local minimizer from that starting point, and extracting the potential value after minimization.

Parameters:

• key (Jax PRNGKey) – A key object used to generate random numbers in Jax. Will be consumed over the running of the function and a new key will be returned.

• minimizer (JaxOpt Projected Gradient optimizer object.) –

• stepper (callable) – The function take_step, wrapped to require only a PRNGKey object as input.

• lam (jnp.array(jnp.float32)) – A 1-D Jax Numpy array representing a set of quartic coefficients in the scalar potential.

Returns:

• energy_after_quench (jnp.float32) – The minimum value of the potential found by minimizer after starting from a random starting position generated with key.

• new_key (Jax PRNGKey) – A key object used to generate random numbers in Jax. Supplied to replace the consumed key.

bfbrain.Jax_Oracle.jax_basinhopping(lam, rng_key, x0, minimizer, niter, polar)

Iterate over run_one_step a large number of times.

Parameters:

• lam (jnp.array(jnp.float32)) – A 1-D Jax Numpy array representing a set of quartic coefficients in the scalar potential.

• rng_key (Jax PRNGKey) – Used to generate random numbers.

• x0 (jnp.array(jnp.float32)) – A Jax NumPy array that tells the function what shape the vev input arrays should be.

• minimizer (JaxOpt Projected Gradient optimizer object.) –

• niter (int) – The maximum number of run_one_step iterations to perform in an attempt to find a negative local minimum of the potential.

• polar (bool) – A flag denoting whether the labeller should use Cartesian coordinates for the vev (False) or convert them into polar coordinates on the phi_len-dimensional unit hypersphere (True).

Returns:

new_min_energy – The smallest local minimum that the optimizer found after its iterations.

Return type:

jnp.float32

bfbrain.Jax_Oracle.vectorized_minTest(func, lam, key, x0, niter, tol, polar)

Vectorized version of vectorized_minTest. Takes similar arguments as vectorized_minTest but with additional array axes over which vectorized_minTest is mapped.

Original documentation:

A vectorized version of jax_basinhopping.

funccallable

A Jax Numpy function that returns the quartic potential and its gradient with respect to the vev parameters. This function will be generated by a BFBrain.DataManager object.

lamjnp.array(jnp.float32, jnp.float32)

A 2-D Jax NumPy array representing multiple sets of quartic coefficients for the potential.

key : Jax PRNGKey

x0jnp.array(jnp.float32)

A Jax NumPy array that informs the function about the shape of the vev input.

niterint

The maximum number of local minimization iterations the minimizer should perform for each set of quartic coefficients in lam, searching for a negative minimum potential value.

toljnp.float32

The tolerance for the local minimizer to stop.

polarbool

A flag denoting whether the labeller should use Cartesian coordinates for the vev (False) or convert them into polar coordinates on the phi_len-dimensional unit hypersphere (True).

jnp.array(jnp.float32)

An array of minimum energy values found for the potentials specified by the elements of lam.

bfbrain.Jax_Oracle.label_func(func, phi_len, polar, rng, lam, niter=100, tol=0.001, cutoff=150000)

The function which interfaces directly with BFBrain.DataManager’s methods for handling oracles.

Parameters:

• func (callable) – The Jax Numpy function that returns the quartic part of the potential and the gradient.

• phi_len (int) – The number of real parameters necessary to uniquely specify a vev.

• polar (bool) – A flag denoting whether the labeller should use Cartesian coordinates for the vev (False) or convert them into polar coordinates on the phi_len-dimensional unit hypersphere (True).

• rng (np.random.Generator) – The NumPy random number generator which will generate the initial PRNGKey used by this oracle.

• lam (np.array(np.float32,np.float32)) – A 2-D Numpy array of quartic potential coefficients.

• niter (int, default=100) – The number of local minimizations to perform on the potential before declaring it to be bounded-from-below.

• tol (float, default=0.001) – The tolerance for the local minimizer.

• cutoff (int, default=150000) – The maximum size of a batch of coefficient values to pass to the GPU at one time. If lam consists of more sets of coefficient values than this, the method will split it into digestible batches.

Returns:

a 1-D NumPy array of labels for each set of quartic coefficients in lam. Labels False for points where the labeller found a negative local minimum and True otherwise.

Return type:

np.array(bool)

bfbrain.Jax_Oracle.label_func_do_batch(func, phi_len, polar, rng, lam, niter, tol)

The method usedy by label_func to transfer the coefficient data to the GPU and perform the jit-compiled analysis with Jax.

Parameters:

• func (callable) – The Jax Numpy function that returns the quartic part of the potential and the gradient.

• phi_len (int) – The number of real parameters necessary to uniquely specify a vev.

• polar (bool) – A flag denoting whether the labeller should use Cartesian coordinates for the vev (False) or convert them into polar coordinates on the phi_len-dimensional unit hypersphere (True).

• rng (np.random.Generator) – The NumPy random number generator which will generate the initial PRNGKey used by this oracle.

• lam (np.array(np.float32,np.float32)) – A 2-D Numpy array of quartic potential coefficients.

• niter (int) – The number of local minimizations to perform on the potential before declaring it to be bounded-from-below.

• tol (float) – The tolerance for the local minimizer.

Returns:

a 1-D NumPy array of labels for each set of quartic coefficients in lam. Labels False for points where the labeller found a negative local minimum and True otherwise.

Return type:

np.array(bool)

bfbrain.Jax_Oracle.test_labeller(func, phi_len, polar, rng, lam, niter=100, tol=0.001, cutoff=150000, niter_step=50, count_success=5, max_iter=20, verbose=False)

A method to test the accuracy of the oracle. Will perform label_func repeatedly for the same 2-D NumPy array of quartic coefficients, but with niter increased each time, until the same labels are returned for for a specified consecutive number of iterations, or some maximum number of labellings has been completed without finding consistent results.

Parameters:

• func (callable) – The Jax Numpy function that returns the quartic part of the potential and the gradient.

• phi_len (int) – The number of real parameters necessary to uniquely specify a vev.

• polar (bool) – A flag denoting whether the labeller should use Cartesian coordinates for the vev (False) or convert them into polar coordinates on the phi_len-dimensional unit hypersphere (True).

• rng (np.random.Generator) – The NumPy random number generator which will generate the initial PRNGKey used by this oracle.

• lam (np.array(np.float32,np.float32)) – A 2-D Numpy array of quartic potential coefficients.

• niter (int, default=100) – The initial number of local minimizations to perform on the potential before declaring it to be bounded-from-below– this value will be incremented over the running of the method.

• tol (float, default=0.001) – The tolerance for the local minimizer.

• cutoff (int, default=150000) – The maximum size of a batch of coefficient values to pass to the GPU at one time. If lam consists of more sets of coefficient values than this, the method will split it into digestible batches.

• niter_step (int, default=50) – The amount to increment the niter parameter of label_func with each successive attempt at labelling lam.

• count_success (int, default=5) – The number of consecutive labelling attempts that must yield identical labels for the function to declare that increasing niter is no longer affecting the results of label_func.

• max_iter (int, default=20) – The maximum number of labelling attempts that the method will make. If no consistent results are found before that time, the test ends in failure.

• verbose (bool, default=False) – If True, print out statements informing the user of the progress of the method.

Returns:

The minimum niter parameter such that count_success consecutive attempts to label lam with increasing niter yielded the same label. If max_iter attempts are made without running into count_success consecutive identical label results, -1 is returned.

Return type:

int

bfbrain.Potential_Functions module

This module contains a number of SymPy functions for the quartic parts of scalar potentials which the program can parse for analysis. Any other potential functions you may want to use can be implemented by writing them in the same format as these.

bfbrain.Potential_Functions.V_Precustodial(phi, lam)

The “precustodial” generalization of the Georgi-Machacheck model given in arXiv:hep-ph/2012.13947. The quartic coefficients are parameterized in the same manner as Eq.(3.4) of that work.

bfbrain.Potential_Functions.V_GM(phi, lam)

The Georgi-Machacheck potential, following the conventions of Eq.(5) of arXiv:hep-ph/1404.2640

bfbrain.Potential_Functions.V_2HDM(phi, lam)

The most general 2-Higgs doublet model, following the conventions of Eq.(1) of hep-ph/0609018

bfbrain.Potential_Functions.V_3HDM(phi, lam)

A 3HDM with Z2xZ2 symmetry, for which no BFB conditions are known.

bfbrain.Potential_Functions.V_3HDM_U1(phi, lam)

A 3HDM with U1xU1 symmetry.

bfbrain.Score_Functions module

This module contains different methods to extract uncertainty estimates from a trained neural network. It also contains two useful analysis functions, MC_call_fast and MC_call_full, which execute Monte Carlo dropout with the neural networks that BFBrain produces.

bfbrain.Score_Functions.MC_call_full(model, lams, n_trials)

Perform predictions on a given input using Monte Carlo dropout. Evaluates the output of model on the input repeatedly, with random dropout applied, and returns all results.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 2-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams. Each entry along the zero axis represents the results of a different forward pass of the same inputs.

Return type:

tf.constant(tf.float32, tf.float32)

bfbrain.Score_Functions.MC_call_fast(model, lams, n_trials)

perform predictions on a given input using Monte Carlo dropout when only the average output is required. Evaluates the output of model on the input lams n_trials times and takes the average output for each point. Schematically equivalent to tf.reduce_mean(MC_call_full(model, lams, n_trials), axis = 0), but faster.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

bfbrain.Score_Functions.QBDC(model, lams, n_trials=100)

Score an ensemble of possible additional training points by “query by dropout committee”. Averages the result of many evaluations with dropout enabled in the network and gives the highest scores to points which are closest to the decision boundary. This should estimate total predictive uncertainty, that is both aleatoric (from ambiguity of the underlying input) and epistemic (from lack of training data in the vicinity of the input) uncertainties.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int, default=100) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

bfbrain.Score_Functions.Max_Entropy(model, lams, n_trials=100)

Score an ensemble of possible additional training points by Shannon entropy. Averages the result of many evaluations with dropout enabled in the network and gives the highest scores to points which have the largest entropy. This should estimate total predictive uncertainty, that is both aleatoric (from ambiguity of the underlying input) and epistemic (from lack of training data in the vicinity of the input) uncertainties.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int, default=100) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

bfbrain.Score_Functions.BALD(model, lams, n_trials=1000)

Score an ensemble of possible additional training points by mutual information (as is done in Bayesian Active Learning by Disagreement, or BALD). This should estimate solely epistemic (stemming from lack of training data in the vicinity of the input) uncertainty.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int, default=1000) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

bfbrain.Score_Functions.Predictive_Variance(model, lams, n_trials=1000)

Score an ensemble of possible additional training points by variance of the predicted score. This should estimate solely epistemic (stemming from lack of training data in the vicinity of the input) uncertainty.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int, default=1000) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

bfbrain.Score_Functions.Variation_Ratios(model, lams, n_trials=1000)

Score an ensemble of possible additional training points by variation ratios (the fraction of evaluations which give the opposite classification to the mode). This should be sensitive to total predictive uncertainty, that is both aleatoric (from ambiguity of the underlying input) and epistemic (from lack of training data in the vicinity of the input) uncertainties

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

• n_trials (int, default=1000) – Specifies the number of forward passes through the neural network to perform when doing Monte Carlo dropout.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

bfbrain.Score_Functions.Random_AL(model, lams)

Score an ensemble of possible additional training points randomly. This can act as a control to confirm that active learning strategies outperform a randomly generated ensemble of training points.

Parameters:

• model (tf.keras.Model) –

• lams (tf.constant(tf.float32, tf.float32)) – A 2-D Tensorflow tensor representing sets of quartic potential coefficients.

Returns:

A 1-D Tensorflow tensor of scores for each set of quartic potential coefficients in lams.

Return type:

tf.constant(tf.float32)

Module contents