ML.NET Series: Membuat Klasifikasi Gambar dengan Transfer Learning (Tensorflow)

Hi Rekan Makers,

Apa kabar ? Sambil menemani waktu libur rekan-rekan kali ini kita coba fitur baru dari ML.NET v1.3 yaitu Image Classification – Preferred API.

Berbeda dari artikel terdahulu disini, tensorflow model hanya digunakan sebagai feature extractor yaitu memroses gambar untuk mengekstrak feature-feature dari gambar, lalu nilai output ini menjadi input untuk layer terakhir dengan algoritma learner multiclass-classification dengan ML.NET seperti sdca, istilahnya lainnya “penultimate”, kita hanya melakukan re-training pada layer terakhir yang dibuat dengan model ML.NET. Sedangkan image classification kali ini kita benar-benar melakukan transfer learning pada pre-trained model tensorflow, jadi akan menghasilkan 2 model output yaitu frozen tensorflow model (.pb) dan model ML.Net (.zip). Saat artikel ini ditulis baru ada 2 pretrained model yang dapat digunakan untuk image classification yaitu ResnetV2101 dan InceptionV3.

Oke sekarang kita coba membuat aplikasi pertama kita, pastikan rekan-rekan minimal sudah menginstall:

Visual Studio 2017 atau versi lebih tinggi / Visual Studio Code
.NET Core v2.2 atau .Net Framework 4.7 ke atas

Langkah berikut menggunakan .NET Core.

Buatlah project baru dengan tipe console, buka terminal / command prompt. Lalu navigasi ke folder yang diinginkan. Ketik “mkdir ImageClassificationMLNet”

ketik “cd ImageClassificationMLNet” untuk masuk ke folder.

ketik “dotnet new console ImageClassificationMLNet” untuk membuat console project.

lalu tambahkan beberapa package

dotnet add package Microsoft.ML -v 1.4.0-preview

dotnet add package Microsoft.ML.Dnn -v 0.16.0-preview

dotnet add package Microsoft.ML.ImageAnalytics -v 1.4.0-preview

dotnet add package SharpZipLib -v 1.2.0

Selanjutnya buka folder tersebut dengan VSCode.

Buka file “Program.cs”, masukan kode berikut:

using System;

using System.Collections;

using System.Collections.Generic;

using System.IO;

using System.Threading.Tasks;

using ImageClassification.Models;

using Microsoft.ML;

using Microsoft.ML.Transforms;

using static Microsoft.ML.DataOperationsCatalog;

using System.Linq;

using Microsoft.ML.Data;

using Helpers;

namespace ImageClassificationMLNet

{

    class Program

    {

        static void Main(string[] args)

        {

            //TrainModel();

            TestModel();

            Console.WriteLine("Press any key to finish");

            Console.ReadKey();

        }

        static void TestModel()

        {

            string imagesForPredictions = GetAbsolutePath(@"../../../predict");

            string savedModelPath = GetAbsolutePath(@"../../../savedmodels");

            var imageClassifierModelZipFilePath = Path.Combine(savedModelPath, "imageClassifier.zip");

            try

            {

                MLContext mlContext = new MLContext(seed: 1);

                Console.WriteLine($"Loading model from: {imageClassifierModelZipFilePath}");

                // Load the model

                ITransformer loadedModel = mlContext.Model.Load(imageClassifierModelZipFilePath, out var modelInputSchema);

                // Create prediction engine to try a single prediction (input = ImageData, output = ImagePrediction)

                var predictionEngine = mlContext.Model.CreatePredictionEngine<ImageData, ImagePrediction>(loadedModel);

                IEnumerable<ImageData> imagesToPredict = LoadImagesFromDirectory(imagesForPredictions, true);

                //Predict the first image in the folder

                ImageData imageToPredict = new ImageData

                {

                    ImagePath = imagesToPredict.First().ImagePath

                };

                var prediction = predictionEngine.Predict(imageToPredict);

                var index = prediction.PredictedLabel;

                // Obtain the original label names to map through the predicted label-index

                VBuffer<ReadOnlyMemory<char>> keys = default;

                predictionEngine.OutputSchema["LabelAsKey"].GetKeyValues(ref keys);

                var originalLabels = keys.DenseValues().ToArray();

                Console.WriteLine($"ImageFile : [{Path.GetFileName(imageToPredict.ImagePath)}], " +

                                  $"Scores : [{string.Join(",", prediction.Score)}], " +

                                  $"Predicted Label : {originalLabels[index]}");

                //Predict all images in the folder

                //

                Console.WriteLine("");

                Console.WriteLine("Predicting several images...");

                foreach (ImageData currentImageToPredict in imagesToPredict)

                {

                    var currentPrediction = predictionEngine.Predict(currentImageToPredict);

                    var currentIndex = currentPrediction.PredictedLabel;

                    Console.WriteLine($"ImageFile : [{Path.GetFileName(currentImageToPredict.ImagePath)}], " +

                                      $"Scores : [{string.Join(",", currentPrediction.Score)}], " +

                                      $"Predicted Label : {originalLabels[currentIndex]}");

                }

            }

            catch (Exception ex)

            {

                Console.WriteLine(ex.ToString());

            }

        }

        static void TrainModel()

        {

            string trainingPath = GetAbsolutePath(@"../../../training");

            string predictPath = GetAbsolutePath(@"../../../predict");

            string savedModelPath = GetAbsolutePath(@"../../../savedmodels");

            var outputMlNetModelFilePath = Path.Combine(savedModelPath, "imageClassifier.zip");

            // 1. Download the image set and unzip

            string finalImagesFolderName = DownloadImageSet(trainingPath);

            string trainingImagesFolder = Path.Combine(trainingPath, finalImagesFolderName);

            MLContext mlContext = new MLContext(seed: 1);

            // 2. Load the initial full image-set into an IDataView and shuffle so it'll be better balanced

            IEnumerable<ImageData> images = LoadImagesFromDirectory(folder: trainingImagesFolder, useFolderNameasLabel: true);

            IDataView fullImagesDataset = mlContext.Data.LoadFromEnumerable(images);

            IDataView shuffledFullImagesDataset = mlContext.Data.ShuffleRows(fullImagesDataset);

            // 3. Split the data 80:20 into train and test sets, train and evaluate.

            TrainTestData trainTestData = mlContext.Data.TrainTestSplit(shuffledFullImagesDataset, testFraction: 0.2);

            IDataView trainDataView = trainTestData.TrainSet;

            IDataView testDataView = trainTestData.TestSet;

            //// OPTIONAL (*1*)  

            // Prepare the Validation set to be used by the internal TensorFlow training process

            // This step is optional but needed if you want to get validation performed while training in TensorFlow

            IDataView transformedValidationDataView = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: "LabelAsKey",

                                                                            inputColumnName: "Label",

                                                                            keyOrdinality: ValueToKeyMappingEstimator.KeyOrdinality.ByValue)

                                                        .Fit(testDataView)

                                                        .Transform(testDataView);

            // 4. Define the model's training pipeline 

            var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: "LabelAsKey",

                                                                            inputColumnName: "Label",

                                                                            keyOrdinality: ValueToKeyMappingEstimator.KeyOrdinality.ByValue)

                        .Append(mlContext.Model.ImageClassification("ImagePath", "LabelAsKey",

                                        arch: ImageClassificationEstimator.Architecture.ResnetV2101,

                                        epoch: 100,     //An epoch is one learning cycle where the learner sees the whole training data set.

                                        batchSize: 30,  // batchSize sets the number of images to feed the model at a time. It needs to divide the training set evenly or the remaining part won't be used for training.                              

                                        metricsCallback: (metrics) => Console.WriteLine(metrics),

                                        //OPTIONAL (*1*) 

                                        validationSet: transformedValidationDataView));

            // 4. Train/create the ML model

            Console.WriteLine("*** Training the image classification model with DNN Transfer Learning on top of the selected pre-trained model/architecture ***");

            ITransformer trainedModel = pipeline.Fit(trainDataView);

            // 5. Get the quality metrics (accuracy, etc.)

            EvaluateModel(mlContext, testDataView, trainedModel);

            // 6. Try a single prediction simulating an end-user app

            TrySinglePrediction(predictPath, mlContext, trainedModel);

            // 7. Save the model to assets/outputs (You get ML.NET .zip model file and TensorFlow .pb model file)

            mlContext.Model.Save(trainedModel, trainDataView.Schema, outputMlNetModelFilePath);

            Console.WriteLine($"Model saved to: {outputMlNetModelFilePath}");

        }

        private static void EvaluateModel(MLContext mlContext, IDataView testDataset, ITransformer trainedModel)

        {

            Console.WriteLine("Making predictions in bulk for evaluating model's quality...");

            IDataView predictionsDataView = trainedModel.Transform(testDataset);

            var metrics = mlContext.MulticlassClassification.Evaluate(predictionsDataView, labelColumnName: "LabelAsKey", predictedLabelColumnName: "PredictedLabel");

            ConsoleHelper.PrintMultiClassClassificationMetrics("TensorFlow DNN Transfer Learning", metrics);

            Console.WriteLine("*** Showing all the predictions ***");

            // Find the original label names.

            VBuffer<ReadOnlyMemory<char>> keys = default;

            predictionsDataView.Schema["LabelAsKey"].GetKeyValues(ref keys);

            var originalLabels = keys.DenseValues().ToArray();

            List<ImagePredictionEx> predictions = mlContext.Data.CreateEnumerable<ImagePredictionEx>(predictionsDataView, false, true).ToList();

            predictions.ForEach(pred => ConsoleWriteImagePrediction(pred.ImagePath, pred.Label, (originalLabels[pred.PredictedLabel]).ToString(), pred.Score.Max()));

        }

        private static void TrySinglePrediction(string imagesForPredictions, MLContext mlContext, ITransformer trainedModel)

        {

            // Create prediction function to try one prediction

            var predictionEngine = mlContext.Model

                .CreatePredictionEngine<ImageData, ImagePrediction>(trainedModel);

            IEnumerable<ImageData> testImages = LoadImagesFromDirectory(imagesForPredictions, true);

            ImageData imageToPredict = new ImageData

            {

                ImagePath = testImages.First().ImagePath

            };

            var prediction = predictionEngine.Predict(imageToPredict);

            // Find the original label names.

            VBuffer<ReadOnlyMemory<char>> keys = default;

            predictionEngine.OutputSchema["LabelAsKey"].GetKeyValues(ref keys);

            var originalLabels = keys.DenseValues().ToArray();

            var index = prediction.PredictedLabel;

            Console.WriteLine($"ImageFile : [{Path.GetFileName(imageToPredict.ImagePath)}], " +

                              $"Scores : [{string.Join(",", prediction.Score)}], " +

                              $"Predicted Label : {originalLabels[index]}");

        }

        public static IEnumerable<ImageData> LoadImagesFromDirectory(string folder, bool useFolderNameasLabel = true)

        {

            var files = Directory.GetFiles(folder, "*",

                searchOption: SearchOption.AllDirectories);

            foreach (var file in files)

            {

                if ((Path.GetExtension(file) != ".jpg") && (Path.GetExtension(file) != ".png"))

                    continue;

                var label = Path.GetFileName(file);

                if (useFolderNameasLabel)

                    label = Directory.GetParent(file).Name;

                else

                {

                    for (int index = 0; index < label.Length; index++)

                    {

                        if (!char.IsLetter(label[index]))

                        {

                            label = label.Substring(0, index);

                            break;

                        }

                    }

                }

                yield return new ImageData()

                {

                    ImagePath = file,

                    Label = label

                };

            }

        }

        public static string DownloadImageSet(string imagesDownloadFolder)

        {

            // get a set of images to teach the network about the new classes

            //SINGLE SMALL FLOWERS IMAGESET (200 files)

            string fileName = "flower_photos_small_set.zip";

            string url = $"https://mlnetfilestorage.file.core.windows.net/imagesets/flower_images/flower_photos_small_set.zip?st=2019-08-07T21%3A27%3A44Z&se=2030-08-08T21%3A27%3A00Z&sp=rl&sv=2018-03-28&sr=f&sig=SZ0UBX47pXD0F1rmrOM%2BfcwbPVob8hlgFtIlN89micM%3D";

            Web.Download(url, imagesDownloadFolder, fileName);

            Compress.UnZip(Path.Join(imagesDownloadFolder, fileName), imagesDownloadFolder);

            return Path.GetFileNameWithoutExtension(fileName);

        }

        public static string GetAbsolutePath(string relativePath)

        {

            FileInfo _dataRoot = new FileInfo(typeof(Program).Assembly.Location);

            string assemblyFolderPath = _dataRoot.Directory.FullName;

            string fullPath = Path.Combine(assemblyFolderPath, relativePath);

            return fullPath;

        }

        public static void ConsoleWriteImagePrediction(string ImagePath, string Label, string PredictedLabel, float Probability)

        {

            var defaultForeground = Console.ForegroundColor;

            var labelColor = ConsoleColor.Magenta;

            var probColor = ConsoleColor.Blue;

            Console.Write("Image File: ");

            Console.ForegroundColor = labelColor;

            Console.Write($"{Path.GetFileName(ImagePath)}");

            Console.ForegroundColor = defaultForeground;

            Console.Write(" original labeled as ");

            Console.ForegroundColor = labelColor;

            Console.Write(Label);

            Console.ForegroundColor = defaultForeground;

            Console.Write(" predicted as ");

            Console.ForegroundColor = labelColor;

            Console.Write(PredictedLabel);

            Console.ForegroundColor = defaultForeground;

            Console.Write(" with score ");

            Console.ForegroundColor = probColor;

            Console.Write(Probability);

            Console.ForegroundColor = defaultForeground;

            Console.WriteLine("");

        }

    }

}

Untuk run project ini butuh C# versi 7.1 ke atas, silakan buka file ImageClassificationMLNet.csproj lalu tambahkan baris dalam tag <LangVersion/> sebagai berikut:

<PropertyGroup>

<OutputType>Exe</OutputType>

<TargetFramework>netcoreapp2.2</TargetFramework>

<LangVersion>7.2</LangVersion>

</PropertyGroup>

Nah project ini butuh beberapa class-class helper dan model silakan download dari repo ini. Salin folder Models dan Helpers ke dalam project kamu.

Lalu pada program.cs, silakan un-comment method “TrainModel();”

lalu jalankan project dengan mengetik:

dotnet restore

dotnet build

dotnet run

Penjelasan Project

Nah pada program.cs, method “TrainModel” untuk melakukan training model, sedangkan method “TestModel” untuk melakukan evaluasi.

Pada method TrainModel langkahnya sama seperti langkah-langkah pada umumnya ketika kita melakukan training dengan data teks. Tapi perhatikan pada baris ini:

IDataView transformedValidationDataView = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: "LabelAsKey",

inputColumnName: "Label",

keyOrdinality: ValueToKeyMappingEstimator.KeyOrdinality.ByValue)

.Fit(testDataView)

.Transform(testDataView);

// 4. Define the model's training pipeline

varpipeline=mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: "LabelAsKey",

inputColumnName: "Label",

keyOrdinality: ValueToKeyMappingEstimator.KeyOrdinality.ByValue)

.Append(mlContext.Model.ImageClassification("ImagePath", "LabelAsKey",

arch: ImageClassificationEstimator.Architecture.ResnetV2101,

epoch: 100, //An epoch is one learning cycle where the learner sees the whole training data set.

batchSize: 30, // batchSize sets the number of images to feed the model at a time. It needs to divide the training set evenly or the remaining part won't be used for training.

metricsCallback: (metrics) =>Console.WriteLine(metrics),

//OPTIONAL (*1*)

validationSet: transformedValidationDataView));

Pada baris ImageClassificationEstimator.Architecture.Resnetv2101 ini bisa diganti dengan model lain seperti InceptionV3. Dan perhatikan juga, disini kita tidak perlu melakukan resize, atau pre-processing gambar sebelum melakukan training, karena semua proses tersebut akan dilakukan semua secara internal. Kita juga perlu memapping Label dalam bentuk string menjadi key dalam bentuk int.

Pada contoh ini kita menggunakan sampel gambar-gambar bunga yang terbagi menjadi 5 jenis : daisy, dandelion, roses, sun flower, tulips. Nama folder penyimpan gambar-gambar training tersebut digunakan sebagai Label. Jadi di dalam class “ModelInput” kita hanya butuh 2 feature yaitu Label (jenis) dan ImagePath (lokasi gambar dalam komputer).

Setelah melakukan training kita bisa melihat ada 2 model yang digenerate yaitu model dengan extension .pb yaitu re-trained tensorflow model dan pada folder “savedmodels” terdapat file .zip yaitu ML.Net model.

Jika kita lihat model tensorflow yang dihasilkan dengan netron akan seperti ini:

Pendekatan native re-training tensorflow model ini punya 2 kelebihan yaitu:

Tensorflow model yang dihasilkan bisa digunakan di platform yang rekan-rekan inginkan seperti android app, keras, tensorflow, java, dsb.
Dengan pendekatan ini proses re-training bisa dilakukan tidak hanya di layer terakhir tapi beberapa layer lainnya sehingga optimasi akurasi dapat dilakukan.

Ketika method TestModel di eksekusi akan memuat model ML.Net (zip) lalu meng-query gambar di dalam folder Predict dan melakukan inference pada setiap gambar.