To be able to perform hyperparameter optimization, I guess one should better set the random seed so we will better compare the models. So I set the random seed in tensorflow through: tf.set_random_...