Welcome to our end-to-end example of distributed image classification algorithm in transfer learning mode. In this demo, we will use the Amazon sagemaker image classification algorithm in transfer learning mode to fine-tune a pre-trained model (trained on imagenet data) to learn to classify a new dataset. In particular, the pre-trained model will be fine-tuned using caltech-256 dataset.
To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on.
Here we set up the linkage and authentication to AWS services. There are three parts to this:
%%time
import boto3
import re
from sagemaker import get_execution_role
role = get_execution_role()
bucket='sagemaker-galaxy' # customize to your bucket
containers = {'us-west-2': '433757028032.dkr.ecr.us-west-2.amazonaws.com/image-classification:latest',
'us-east-1': '811284229777.dkr.ecr.us-east-1.amazonaws.com/image-classification:latest',
'us-east-2': '825641698319.dkr.ecr.us-east-2.amazonaws.com/image-classification:latest',
'eu-west-1': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/image-classification:latest'}
training_image = containers[boto3.Session().region_name]
print(training_image)
The caltech 256 dataset consist of images from 257 categories (the last one being a clutter category) and has 30k images with a minimum of 80 images and a maximum of about 800 images per category.
The image classification algorithm can take two types of input formats. The first is a recordio format and the other is a lst format. Files for both these formats are available at http://data.dmlc.ml/mxnet/data/caltech-256/. In this example, we will use the recordio format for training and use the training/validation split specified here.
import os
import urllib.request
import boto3
def download(url):
filename = url.split("/")[-1]
if not os.path.exists(filename):
urllib.request.urlretrieve(url, filename)
def upload_to_s3(channel, file):
s3 = boto3.resource('s3')
data = open(file, "rb")
key = channel + '/' + file
s3.Bucket(bucket).put_object(Key=key, Body=data)
# # caltech-256
download('https://s3-us-west-2.amazonaws.com/learn-galaxies/galaxies-train2.rec')
download('https://s3-us-west-2.amazonaws.com/learn-galaxies/galaxies-test2.rec')
upload_to_s3('validation', 'galaxies-test2.rec')
upload_to_s3('train', 'galaxies-train2.rec')
Once we have the data available in the correct format for training, the next step is to actually train the model using the data. Before training the model, we need to setup the training parameters. The next section will explain the parameters in detail.
There are two kinds of parameters that need to be set for training. The first one are the parameters for the training job. These include:
Apart from the above set of parameters, there are hyperparameters that are specific to the algorithm. These are:
After setting training parameters, we kick off training, and poll for status until training is completed, which in this example, takes between 10 to 12 minutes per epoch on a p2.xlarge machine. The network typically converges after 10 epochs.
# The algorithm supports multiple network depth (number of layers). They are 18, 34, 50, 101, 152 and 200
# For this training, we will use 18 layers
num_layers = 101
# we need to specify the input image shape for the training data
image_shape = "3,224,224"
# we also need to specify the number of training samples in the training set
# for caltech it is 15420
num_training_samples = 19*4
# specify the number of output classes
num_classes = 5
# batch size for training
mini_batch_size = 21
# number of epochs
epochs = 5
# learning rate
learning_rate = 0.0018
top_k=2
# Since we are using transfer learning, we set use_pretrained_model to 1 so that weights can be
# initialized with pre-trained weights
use_pretrained_model = 1
Run the training using Amazon sagemaker CreateTrainingJob API
%%time
import time
import boto3
from time import gmtime, strftime
s3 = boto3.client('s3')
# create unique job name
job_name_prefix = 'sagemaker-imageclassification-notebook'
timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())
job_name = job_name_prefix + timestamp
training_params = \
{
# specify the training docker image
"AlgorithmSpecification": {
"TrainingImage": training_image,
"TrainingInputMode": "File"
},
"RoleArn": role,
"OutputDataConfig": {
"S3OutputPath": 's3://{}/{}/output'.format(bucket, job_name_prefix)
},
"ResourceConfig": {
"InstanceCount": 1,
"InstanceType": "ml.p2.xlarge",
"VolumeSizeInGB": 50
},
"TrainingJobName": job_name,
"HyperParameters": {
"image_shape": image_shape,
"num_layers": str(num_layers),
"num_training_samples": str(num_training_samples),
"num_classes": str(num_classes),
"mini_batch_size": str(mini_batch_size),
"epochs": str(epochs),
"learning_rate": str(learning_rate),
"use_pretrained_model": str(use_pretrained_model)
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 360000
},
#Training data should be inside a subdirectory called "train"
#Validation data should be inside a subdirectory called "validation"
#The algorithm currently only supports fullyreplicated model (where data is copied onto each machine)
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": 's3://{}/train/'.format(bucket),
"S3DataDistributionType": "FullyReplicated"
}
},
"ContentType": "application/x-recordio",
"CompressionType": "None"
},
{
"ChannelName": "validation",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": 's3://{}/validation/'.format(bucket),
"S3DataDistributionType": "FullyReplicated"
}
},
"ContentType": "application/x-recordio",
"CompressionType": "None"
}
]
}
print('Training job name: {}'.format(job_name))
print('\nInput Data Location: {}'.format(training_params['InputDataConfig'][0]['DataSource']['S3DataSource']))
# create the Amazon SageMaker training job
sagemaker = boto3.client(service_name='sagemaker')
sagemaker.create_training_job(**training_params)
# confirm that the training job has started
status = sagemaker.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']
print('Training job current status: {}'.format(status))
try:
# wait for the job to finish and report the ending status
sagemaker.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=job_name)
training_info = sagemaker.describe_training_job(TrainingJobName=job_name)
status = training_info['TrainingJobStatus']
print("Training job ended with status: " + status)
except:
print('Training failed to start')
# if exception is raised, that means it has failed
message = sagemaker.describe_training_job(TrainingJobName=job_name)['FailureReason']
print('Training failed with the following error: {}'.format(message))
training_info = sagemaker.describe_training_job(TrainingJobName=job_name)
status = training_info['TrainingJobStatus']
print("Training job ended with status: " + status)
If you see the message,
Training job ended with status: Completed
then that means training sucessfully completed and the output model was stored in the output path specified by training_params['OutputDataConfig'].
You can also view information about and the status of a training job using the AWS SageMaker console. Just click on the "Jobs" tab.
A trained model does nothing on its own. We now want to use the model to perform inference. For this example, that means predicting the topic mixture representing a given document.
This section involves several steps,
We now create a SageMaker Model from the training output. Using the model we can create an Endpoint Configuration.
%%time
import boto3
from time import gmtime, strftime
sage = boto3.Session().client(service_name='sagemaker')
model_name="galaxy-clasifier4"
print(model_name)
info = sage.describe_training_job(TrainingJobName=job_name)
model_data = info['ModelArtifacts']['S3ModelArtifacts']
print(model_data)
containers = {'us-west-2': '433757028032.dkr.ecr.us-west-2.amazonaws.com/image-classification:latest',
'us-east-1': '811284229777.dkr.ecr.us-east-1.amazonaws.com/image-classification:latest',
'us-east-2': '825641698319.dkr.ecr.us-east-2.amazonaws.com/image-classification:latest',
'eu-west-1': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/image-classification:latest'}
hosting_image = containers[boto3.Session().region_name]
primary_container = {
'Image': hosting_image,
'ModelDataUrl': model_data,
}
create_model_response = sage.create_model(
ModelName = model_name,
ExecutionRoleArn = role,
PrimaryContainer = primary_container)
print(create_model_response['ModelArn'])
At launch, we will support configuring REST endpoints in hosting with multiple models, e.g. for A/B testing purposes. In order to support this, customers create an endpoint configuration, that describes the distribution of traffic across the models, whether split, shadowed, or sampled in some way.
In addition, the endpoint configuration describes the instance type required for model deployment, and at launch will describe the autoscaling configuration.
from time import gmtime, strftime
timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())
endpoint_config_name = job_name_prefix + '-epc-' + timestamp
endpoint_config_response = sage.create_endpoint_config(
EndpointConfigName = endpoint_config_name,
ProductionVariants=[{
'InstanceType':'ml.m4.xlarge',
'InitialInstanceCount':1,
'ModelName':model_name,
'VariantName':'AllTraffic'}])
print('Endpoint configuration name: {}'.format(endpoint_config_name))
print('Endpoint configuration arn: {}'.format(endpoint_config_response['EndpointConfigArn']))
Lastly, the customer creates the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications. This takes 9-11 minutes to complete.
%%time
import time
timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())
endpoint_name = job_name_prefix + '-ep-' + timestamp
print('Endpoint name: {}'.format(endpoint_name))
endpoint_params = {
'EndpointName': endpoint_name,
'EndpointConfigName': endpoint_config_name,
}
endpoint_response = sagemaker.create_endpoint(**endpoint_params)
print('EndpointArn = {}'.format(endpoint_response['EndpointArn']))
Finally, now the endpoint can be created. It may take sometime to create the endpoint...
# get the status of the endpoint
response = sagemaker.describe_endpoint(EndpointName=endpoint_name)
status = response['EndpointStatus']
print('EndpointStatus = {}'.format(status))
# wait until the status has changed
sagemaker.get_waiter('endpoint_in_service').wait(EndpointName=endpoint_name)
# print the status of the endpoint
endpoint_response = sagemaker.describe_endpoint(EndpointName=endpoint_name)
status = endpoint_response['EndpointStatus']
print('Endpoint creation ended with EndpointStatus = {}'.format(status))
if status != 'InService':
raise Exception('Endpoint creation failed.')
If you see the message,
Endpoint creation ended with EndpointStatus = InService
then congratulations! You now have a functioning inference endpoint. You can confirm the endpoint configuration and status by navigating to the "Endpoints" tab in the AWS SageMaker console.
We will finally create a runtime object from which we can invoke the endpoint.
Finally, the customer can now validate the model for use. They can obtain the endpoint from the client library using the result from previous operations, and generate classifications from the trained model using that endpoint.
import boto3
runtime = boto3.Session().client(service_name='runtime.sagemaker')
import cv2
#!wget https://s3-us-west-2.amazonaws.com/learn-galaxies/bigspiral.zip
#!unzip bigspiral.zip
#!wget https://s3-us-west-2.amazonaws.com/learn-galaxies/bigtest.zip
#!unzip bigtest.zip
#!wget https://s3-us-west-2.amazonaws.com/learn-galaxies/bigelliptical.zip
#!unzip bigelliptical.zip
#!wget https://s3-us-west-2.amazonaws.com/learn-galaxies/bigbarred.zip
#!unzip bigbarred.zip
#!wget https://s3-us-west-2.amazonaws.com/learn-galaxies/bigirregular.zip
#!unzip bigirregular.zip
file_name = 'bigspiral/s1.jpg'
# test image
from IPython.display import Image
Image(file_name)
import json
import numpy as np
with open(file_name, 'rb') as f:
payload = f.read()
payload = bytearray(payload)
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
ContentType='application/x-image',
Body=payload)
result = response['Body'].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
print(result)
index = np.argmax(result)
print(index)
object_categories = ['unk', 'barred spiral', 'elliptical', 'irregular', 'spiral']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))
predictions = []
for i in range(0,40):
file_name = 'bigtest/t'+str(i)+'.jpg'
with open(file_name, 'rb') as f:
payload = f.read()
payload = bytearray(payload)
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
ContentType='application/x-image',
Body=payload)
result = response['Body'].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
print(result)
index = np.argmax(result)
print(index)
object_categories = ['unk', 'barred spiral', 'elliptical', 'irregular', 'spiral']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))
predictions.append(index)
mydict = {'spiral': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0},
'barred spiral': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0},
'elliptical': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0},
'irregular': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0}}
keys = []
for i in range(4):
for j in range(10):
keys.append(i+1)
keytab = {1: 'barred spiral', 2: 'elliptical', 3:'irregular', 4:'spiral'}
i = 0
for key in keys:
mydict[keytab[key]][keytab[predictions[i]]]+= 1
i = i+1
import pandas as pd
df = pd.DataFrame.from_dict(mydict)
df.transpose()
train_predictions = []
for i in range(1,20):
file_name = 'bigbarred/bs'+str(i)+'.jpg'
with open(file_name, 'rb') as f:
payload = f.read()
payload = bytearray(payload)
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
ContentType='application/x-image',
Body=payload)
result = response['Body'].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
print(result)
index = np.argmax(result)
print(index)
object_categories = ['unk', 'barred spiral', 'elliptical', 'irregular', 'spiral']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))
train_predictions.append(index)
print("now elliptical")
for i in range(1,20):
file_name = 'bigelliptical/e'+str(i)+'.jpg'
with open(file_name, 'rb') as f:
payload = f.read()
payload = bytearray(payload)
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
ContentType='application/x-image',
Body=payload)
result = response['Body'].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
print(result)
index = np.argmax(result)
print(index)
object_categories = ['unk', 'barred spiral', 'elliptical', 'irregular', 'spiral']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))
train_predictions.append(index)
print("now irregular")
for i in range(1,20):
file_name = 'bigirregular/i'+str(i)+'.jpg'
with open(file_name, 'rb') as f:
payload = f.read()
payload = bytearray(payload)
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
ContentType='application/x-image',
Body=payload)
result = response['Body'].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
print(result)
index = np.argmax(result)
print(index)
object_categories = ['unk', 'barred spiral', 'elliptical', 'irregular', 'spiral']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))
train_predictions.append(index)
print("now spiral")
for i in range(1,20):
file_name = 'bigspiral/s'+str(i)+'.jpg'
with open(file_name, 'rb') as f:
payload = f.read()
payload = bytearray(payload)
response = runtime.invoke_endpoint(EndpointName=endpoint_name,
ContentType='application/x-image',
Body=payload)
result = response['Body'].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
print(result)
index = np.argmax(result)
print(index)
object_categories = ['unk', 'barred spiral', 'elliptical', 'irregular', 'spiral']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))
train_predictions.append(index)
len(train_predictions)
train_keys = []
for i in range(4):
for j in range(19):
train_keys.append(i+1)
train_mydict = {'spiral': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0},
'barred spiral': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0},
'elliptical': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0},
'irregular': {'spiral':0.0, 'barred spiral':0.0, 'elliptical':0.0, 'irregular':0.0}}
keytab = {1: 'barred spiral', 2: 'elliptical', 3:'irregular', 4:'spiral'}
i = 0
for key in train_keys:
train_mydict[keytab[key]][keytab[train_predictions[i]]]+= 1
i = i+1
train_df = pd.DataFrame.from_dict(train_mydict)
train_df.transpose()
When we're done with the endpoint, we can just delete it and the backing instances will be released. Run the following cell to delete the endpoint.
sage.delete_endpoint(EndpointName=endpoint_name)