Developing, training, and testing deep learning models is all done using a single platform, whether it is TensorFlow, PyTorch, Keras, MATALB or one of the many platforms out there. But in some cases, it is required to migrate models between different platforms. One such example is migrating a model from a complete training framework (such as TensorFlow, PyTorch, etc.) to a lite inference only framework (such as TensorRT, OpenCV Deep Neural Networks, etc.). Another recurring example comes from the practice of transfer learning: in many cases a desired pre-trained model is available only in certain frameworks and using it (without replacing your existing framework) will require migration of the model between different frameworks.
Importing and exporting models boils down to finding the right format that both frameworks can read and write. Such formats may include the Keras H5 format, TensorFlow SavedModel format, ONNX, or others. Once a common format is found, nonstandard layers will have to be manually converted to allow correct implementation on both frameworks. Finally, the network inference will be tested in both frameworks. In this blog we will describe an example for such migration process along with code examples.
Migrating a TensorFlow 2.1 model to OpenCV DNN (Deep Neural Networks):
It is common for a production code to run on a native C++, so the ability to perform model inference using OpenCV instead of TensorFlow (TF) will result in a much simpler integration and will improve run time. Unfortunately TF 2.1 default format (SavedModel format) is currently not supported by OpenCV DNN, but a rather simple solution exists: by converting all of the network weights to constants (freezing) it is possible to save the entire model in a single *.pb file supported by OpenCV DNN (similar to the *.pb format in TF 1.x)[i].
For example, using this procedure we can export a pre-trained resNet50 model from TF to OpenCV.
(Full code and environment can be found at https://github.com/orinoked1/TF2_to_OpenCV)
Loading the pre-trained network:
import tensorflow as tf
from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2
import os
# get a pre-trained ResNet50 model
base_model = tf.keras.applications.ResNet50(weights='imagenet')
model_input_shape = base_model.input_shape
Saving it in TF native SavedModel format locally:
# save model in SavedModel format
SavedModel_path = os.path.join('model', 'SavedModel_folder')
base_model.save(SavedModel_path, save_format='tf')Saving it as a frozen graph:
# load saved model and freeze it
loaded_model = tf.saved_model.load(SavedModel_path)
infer = loaded_model.signatures['serving_default']
concrete_function = tf.function(infer).get_concrete_function(input_1=tf.TensorSpec(shape=model_input_shape,
dtype=tf.float32))
wrapped_fun = convert_variables_to_constants_v2(concrete_function)
graph_def = wrapped_fun.graph.as_graph_def()
# save again in a *.pb file including weighs as constants
model_frozen_pb_full_path = os.path.join('model', 'frozenGraph_folder')
os.makedirs(model_frozen_pb_full_path, exist_ok=True)
with tf.io.gfile.GFile(os.path.join(model_frozen_pb_full_path,'frozenGraph.pb'), 'wb') as f:
f.write(graph_def.SerializeToString())Then we can perform inference on the same random input in both frameworks (notice that in TF the input is shape is channels first while in DNN it is channels last):
import cv2 as cv
import numpy as np
# generate random input
image_shape = np.asarray(model_input_shape[1:])
input_shape = np.insert(image_shape, 0, 1, axis=0) # add observation dim
test_input_c_last = np.random.standard_normal(input_shape).astype(np.float32)
test_input_c_first = np.moveaxis(test_input_c_last, -1, 1)
# inference in tensorflow
out_tf = base_model.predict(test_input_c_last)
# inference in openCV DNN
net = cv.dnn.readNet(os.path.join(model_frozen_pb_full_path,'frozenGraph.pb'))
net.setInput(test_input_c_first)
out_dnn = net.forward()and finally compare the two feature vectors :
# compare feature vectors
np.testing.assert_allclose(out_tf, out_dnn, rtol=1e-03, atol=1e-05)
max_abs_diff = np.max(np.abs(out_tf-out_dnn))
max_rel_diff = np.max(np.abs((out_tf-out_dnn)/out_tf))
print('max absolute difference is {:e} max relative difference is {:e}'.format(max_abs_diff,max_rel_diff))Next we can use OpenCV to perform inference in a C++ project, first we will save the TF model results along with the randomly generated input image:
# save TF result to compare to Cpp
test_image_folder = 'img_folder'
os.makedirs(test_image_folder, exist_ok=True)
np.save(os.path.join(test_image_folder, 'TF_feature_vector.npy'), out_tf)
# export the test image to a Cpp project (using openCV *.xml file format)# save the float array with shape [rows X cols X channels]
fs = cv.FileStorage(os.path.join(test_image_folder, 'test_img.xml'), cv.FILE_STORAGE_WRITE)
fs.write("test_img", np.squeeze(test_input_c_last))
fs.release()
Then we can open a visual studio project to perform inference in C++.
#include <windows.h>#include <opencv2/dnn.hpp>
using namespace cv;
using namespace std;
using namespace dnn;
int main(int argc, char** argv)v{ // define net path & test file path סססססססססססססstring netPath = "C:\\git_repos\\TF2_to_OpenCV\\py\\model\\frozenGraph_folder\\frozenGraph.pb"; ססססססססססססס// load network סססססססססססססNet resNet50 = readNet(netPath); ססססססססססססס// load test image סססססססססססססstring inputFilePath = "C:\\git_repos\\TF2_to_OpenCV\\py\\img_folder\\test_img.xml"; סססססססססססססFileStorage inputFile; סססססססססססססMat inputImage; ססססססססססססס inputFile.open(inputFilePath, FileStorage::READ); סססססססססססססinputFile["test_img"] >> inputImage; סססססססססססססinputFile.release(); ססססססססססססס// reshape image to blob [rows X cols X channels] -> [observation(1) X channels X rows X cols] סססססססססססססMat blob; ססססססססססססס blobFromImage(inputImage, blob, 1.0, Size(inputImage.cols, inputImage.rows), Scalar(0.0), false, false); ססססססססססססס// forward pass סססססססססססססMat featureVector; סססססססססססססresNet50.setInput(blob); סססססססססססססfeatureVector = resNet50.forward(); ססססססססססססס// save output ססססססססססססס string outputFilePath = "C:\\git_repos\\TF2_to_OpenCV\\py\\img_folder\\cpp_feature_vector.xml"; סססססססססססססFileStorage outputFile; סססססססססססססoutputFile.open(outputFilePath, FileStorage::WRITE); סססססססססססססoutputFile << "feature_vector" << featureVector; סססססססססססססoutputFile.release(); }
And finally, compare the C++ feature vector to the TF feature vector:
import cv2 as cv
import numpy as np
import os
test_image_folder = 'img_folder'
tf_out = np.load(os.path.join(test_image_folder, 'TF_feature_vector.npy'))
openCV_file = cv.FileStorage(os.path.join(test_image_folder,'cpp_feature_vector.xml'),
cv.FILE_STORAGE_READ)
openCV_out = openCV_file.getFirstTopLevelNode().mat()
np.testing.assert_allclose(tf_out, openCV_out, rtol=1e-03, atol=1e-05)
max_abs_diff = np.max(np.abs(tf_out-openCV_out))
max_rel_diff = np.max(np.abs((tf_out-openCV_out)/tf_out))
print('max absolute difference is {:e} max relative difference is {:e}'.format(max_abs_diff,max_rel_diff))