HomeAI TrainingTensorFlow DemosMini Batches

TensorFlow Demos

Image Classification with ResNet50Text Extraction with BERTBidirectional LSTM

Bidirectional LSTM

To demonstrate the performance of the B67 algorithm vs. Adam, a bidirectional LSTM neural network was trained on a random-number data set using both algorithms (see Network and Data Set).

B67's loss value after nine epochs was matched by that of the Adam algorithm after 90 epochs (see below). B67 ran for 23 minutes vs. 100 seconds for Adam due to the performance limitation incurred by TensorFlow's constraints on our basic demo implementation (see Performance Penalty). In a production environment, this performance penalty does not exist and the time per epoch would be comparable to Adam.

Result Summary

Results are shown below. A plot of loss vs. time is omitted as it is not representative of performance (see Performance Penalty).

Convergence

Raw Data

Summary logs and screen output are available.

Demonstration Limitations

Performance Penalty

For this demonstration, B67 is implemented as a custom TensorFlow 'op.' The demonstration runs significantly more slowly than it would in a production environment due to:

  1. Recalculation of values that TensorFlow has already performed, since the framework does not supply B67's required inputs by default. This recalculation restricts B67 to performing a step once every two epochs.
  2. CPU implementation, which is many times slower than running calculations on GPU. In a production environment, a B67 epoch would take roughly the same amount of time as an Adam epoch or, with recalculations in (1) performed, roughly the time for two Adam epochs.

Test Setup

Network and Data Set

The network used is the one described in this Keras example without the sigmoid activation on the output node. Since the Keras example converges quickly, we use a random-number data set to better demonstrate relative performance:

Randomness Minimized for Performance Comparison

The random seeds for Python, Python's numpy library, and TensorFlow are set to fixed values. In addition, shuffling of data sets is disabled.

Note that there still exist unavoidable variations in floating-point values due to non-deterministic ordering of parallel operations and aggregation of GPU calculations.

Loss function, batch size

  1. The loss function used is: .
  2. The batch size is 3500 samples (largest possible with available memory).

Adam

Adam was run with default parameters.

Software

Hardware