Hey, guys. Hey, guys. Hello. How are you doing? Hey, Nilesh. How are you? Hey, Ram. I guess people are still busy. Yeah. But they'll come online. How are you doing? Doing great. Cool. So can you see my desktop? Yeah, I can. Can you see me? Yes. OK. So what I was wondering while other people come along, I think it's just you and me right now. Can you talk about the thing that you had in mind about Rails? Sure. Right. So what I did specifically for after we talked, after you mentioned Rails, I put together a Git repository that basically might be what you need in terms of getting Rails application on Docker. And that is the thing that I think I gave out in that this is open source now. OK. And you should see that link that I will post on Slack chat so you have it there. So pasting it. And you should have it already, by the way. There is a link I gave on this time, like the other day, that I have created a new access exercise for that, which is right here. So this exercise basically helps you construct a new Rails application and scale it in Docker. Got it. OK. It's nerdy. But you're OK with nerdiness. So this is actually creating some new application. And how do you begin? Right. OK. At the end of the exercise, what happens is let me quickly show it to you. So if you have the exercise done, at the end of the exercise, you will have, let's see, we have it here. Yes. So OK. No, no, no. We don't have it here. Desktop. And what do we have? Docker, CT Rails, then Cloud Genius, and Docker Compose. Up. That's what we have. So at the end of the exercise, that's what it will look like. OK. And then you will open localhost. And you will find that it is giving an error because the database is starting up and listening on 8,000. The database is still starting. So that may be the reason why it is giving an error. But don't worry. It will stop. And this error is documented. This is expected, you are expected to see this error, by the way. This error is that could not connect to server. The database is not available. So that error is expected. OK. However you run it first time, that's what you saw. So we will basically follow along in that. So you have to initialize the database by running this command. This will initialize the database itself, which basically is what I'm doing right now. And this is going to reset the DV. And let's see what's going on. Why is it not responding quickly? OK. Could not drop the database, previous database. I could not drop test. Apparently, I need to clean up. So there are some previous runs that I'm doing, apparently, that needs to be cleaned up. So let me clean up. OK. Clean that up nicely. So here, docker rm-f, docker psaq. And so keeping all of those containers, then we will do a reset. So we can reset. And at that time, it will begin a fresh database and allow us to connect once we have a migration completed. That's OK. It will still give in the error. So forget the error that we are running. It will not give you error on your attempt. Got it. OK. I have messed up my machine. Sorry. That always happens. You give me errors. I'm fine with that. Yeah. Now, we will do a migrate. And so what this exercise basically does is, if you look at my screen, I'm going to draw the illustration. The illustration would be something like, so a couple of things. We have a Rails container, first thing. Rails will require some database. So we have a database container for Postgres. That's another container. Right? Right. Sometimes, we need some worker process. Yes. So worker process, we have another container using sidekick. Sidekick is a separate container. And then, let's see if the Docker Compose file should show me all that. So I'm opening it up right now in Docker Compose. That file will show us what tab would we have. So Postgres is one container, Redis is another that I did not mention. This Redis is used to store information about what the work is. So the worker is working. And so this Redis container will store information about what work items are kept here. So it's like a DB to store whatever the worker's work items are. That's a separate container. This database is primarily used by the application itself. And then, we will use the same Redis for caching some of our content for the application. So we'll have Redis used multiple places. This will use Redis. This will also use Redis. Yeah. And that is this Redis image. Then we have the application itself. I'm calling it the Cloud Genius application. We have a sidekick running there. That's the sidekick container. And that's it. I think there are four containers, so one, two, three, and four. These are defined in your Git repository. You should have those already defined here. So there is a version one Docker Compose, which is the older one. And this is a newer version of Docker Compose that defines those one, two, three, and four containers. On a clean, new machine, they should work. I would not begin looking at the Git repository quite right to begin with. I would actually begin here and build it yourself. I would like to build the repository yourself. That's what I would do. I would not go here. Yeah, that gives a deeper understanding of how exactly... Yes. So this describes how to build it. That will help you in... And once you get to the point where you can get to a situation where you have the initial one ready, what I would do is tell you a method where you can do a continuous integration I think that part is the most interesting part of this game. So this continuous integration that you did not mention, but I think you need that. What I would like to see is I'll let you play with this, not this Git repository, but build your own Git repository based on this exercise. Sure. Yeah. Once you have that, you will have something like a simple Rails application running. Once you have that running, what I would do is then use continuous integration with Docker Cloud and tell you how that works. So that will help you in understanding this a little better. So the idea behind continuous integration would be something like this. That you have some application that you developed and it requires one container or multiple containers it might need and they all configure connect together through the compose file. This compose file will then generate your application. What you need to be doing is put all of this inside a Git repository and ideally push it to some place where like GitHub, for example. That's where you push and it lives there until you change something here. You make changes here. You need to then push it again. Right. And we're talking about this concept called continuous integration. Enable that continuous integration, meaning all I will do in my life is change and push, change and push. That's what I really want to do. Everything else should be automated. That's the concept behind continuous integration. You know this. Right. You understand this. Yes. So what I want to be able to do is, is somehow have your app runs on your laptop in bunch of containers. So that's great. You put this inside a Git repository, which is nice. Then you have this in GitHub or Bitbucket doesn't matter what you have it somewhere. I was just a bit late, you know, there was this update run on my computer and the internet couldn't connect. So finally figured out. No worries. No worries. I just saw you there. Hi. And so I was, while it was just Ram and me, so I was talking about what Ram was interested in Rails and things like that. So I was discussing that briefly and I... That's good stuff. Keep going. Okay. Okay. Yeah. It was specific requests that Ram had the other day and that's what I was talking about. So let me just finish that part and we'll come back to it as appropriate. So the idea is that you have this app that you can build based on this exercise that is here. That's a new exercise I added the other day, two days ago. And what you will do is you will then prepare your app. This is the sample. This is a sample application, right? Not too complex. It's pretty much simple, very simple sample. And then you want to basically look at continuous integration. So that's what I'm talking about. So you have the app, you make changes, and then you push. You push it to Bitbucket, push it to GitHub, wherever you feel like. That's great. Now, how does CI work? The idea behind CI is that there has to be some build engine or some other place where you will actually build fresh images, fresh images out of this repository. Like Jenkins. Yeah. So Jenkins is one good example. So there has to be a place where you build something based on whatever code you create. You create code here, like in your laptop, and you push it to Bitbucket. Great. Now, there's new code here, new, but the images are old. So we need to build new images, right? And that essentially is the first step of continuous integration. You have to build new images, fresh images. And then the next logical step is to use those images in production. The production usage, if that's the scenario, then how do we put things together? How do we connect things together? That's the next logical question. Like you can use Jenkins. There are a hundred other places to solve. The simplest way to begin understanding this concept is in Docker Cloud, cloud.docker.com. That's the easiest way to understand it, I think, in the context of Docker, by the way. So cloud.docker.com. If you go here and then you sign up, and maybe you sign up with something that actually works about this, log in with Docker ID and log in. And there we go. So we are now inside Docker Cloud. And now, by the way, what is Docker Cloud? That's the question that many people have. Is it a cloud? No, it is not a cloud. It is not at all. It is a service. What does it do? It connects to a variety of clouds. And whatever you feel like, your choice of cloud, like DigitalOcean, AWS, Azure, things like that, and many others. And you can bring your own cloud, BYOC, and so connect that. And then this service will help you manage containers in your cloud. Any of these. It basically helps you manage containers. So what does that do? You begin with linking a provider. So you click. You link some providers, like I have connected Amazon, DigitalOcean, Azure, and software, and so on. So some of them are connected, some of them are not. Like this one, relink provider. I'll just link it up again, and so it will go to DigitalOcean and then connect it. So this DigitalOcean is connected now. Azure, I have a credential included here that basically I can change if I like, and it is connected. This one is not connected. Software. I don't use that cloud much. This one is connected. So the packet cloud is connected. Amazon, I have disconnected. So I can always connect. I can add new credentials by basically editing here and making sure that it gets connected. So these are the places that I can connect my clouds. Then I have also provided some providers. This concept of provider basically helps me bring my code into the Docker cloud. So here, that's the connection. I have a GitHub connection, I have a Bitbucket connection. Next what I will do is hook it up with Slack. This is optional, by the way. This is just a nice thing. You have to hook up with some notification methods so you can get notifications in email or get notifications in Slack. I prefer Slack, so I'll use that, and billing information, all that. So what did I describe here? The idea that I'm doing basically is that you have this service called Docker Cloud that connects to GitHub or Bitbucket. It also connects to a variety of cloud providers of your choice, and it helps you manage containers. That's what I just said. What it also does is that any time you do a push to Bitbucket, and you do a commit, and then you do a push, this push is picked up by a push, by a hook, you have to create a hook. The hook gets connected between your Bitbucket application that runs on Bitbucket company, the Classium company, and this hook will basically prompt Docker Cloud to tell Docker Cloud that there is a push. And that's how it picks up new code. Once you have this method included in form of a hook that you can hook either with GitHub or Bitbucket, it will then prompt this as soon as there is a hook, and you push, it generates some kind of a signal to this place, Docker Cloud, that there is new code available. And so once that new code signal becomes available to Docker Cloud, it kicks off a build cycle. That build cycle gets picked up by some engine that will actually pull your code changes and basically run Docker build. It will build that new code that you pushed. And as a consequence of this build process, there is a new Docker image created. This Docker image gets saved in the Docker registry automatically. And as soon as you have a new image ready that works successfully, that build passes, it will get deployed by the Docker Cloud. So your actual cloud solution that you may be running on this cloud or that cloud or that cloud, it will send the new image over to these places and automatically restart those services on live production. And until this new Docker gets going, the previous Docker will keep running. Previously, it runs with the old version, the new one, then it kills, the new one starts. That's it. It's as simple as that. If there is interest in this idea, I can demonstrate that. Let's see how many people are just two. So let me ask both of you. Is this interesting? I can maybe do a live very quick, simple demonstration so you understand what's the mechanics of it. I can begin by creating a simple application. So let's see. So we will on my desktop, somewhere here, I will create a new folder. So new folder, let's call it continuous integration, CI. And go inside that folder, CI. And then open the editor. So very simple, almost, at this moment, the CI folder does not contain anything. So I'm beginning from basically scratch. Now I will create something new, some new application right here. And I will call that application, we'll call this file Docker file. In that file, I will say from Ubuntu. And then inside there, I will copy version, a file called version into the root folder of the machine, the image that I create. And then that's it. That's what I have. So that's my version. That's the only change I want to do. And what I will also do is version number one is the version that I will create a file called version, which says version number is, version number is number one. That's the file. It is going to be copied over to the root location inside the image. That's my Docker file. So now it is easy to build this. What I will do now is go to the terminal and build, Docker build, and give it a name. So I will call it dash d cloud genius slash ci and dot. Use my current folder to build a new image called cloud genius slash ci. And it starts to build. It says I'm going to use the Ubuntu foundation. So that's the foundation. And I'm going to put like a copy a new file inside there. The file will be called version into the root folder. That's what I just did. Line number three. And that's what happened here. Step number two, copy this file into the root folder. And so now we have that image ready. This image is ready. What I want to be able to do is make sure that I actually run this for real in a cloud. So first of all, before running it in the cloud, I want to run it here. So something like this, Docker. And then execute or rather run interactively and give me a teletype writer and cloud genius and then ci and give me a batch prompt inside. And I have it. And there I list and I go to this root folder, but I have that file and I cat it and it's there. So this version is number one. Great. That's my version number one application running, as you can see, very trivially simple application that is running right now in a Docker container in my local laptop. I come out of it. So I have this thing available. I'm going to put this in a Git repository and then Git add and then Git commit. And then I will say initial version one. And then I will like to send this to GitHub or to BitBudget, whatever you prefer. So I'll say GitHub and in there I will create a new repository. The repository is called ci and it is public. So here we go, create one. And then I will add that link over to here, the Git remote, and then I can push. So once I push, my repository is available in this location. So now my repository is available right in that location in GitHub. So I can clone it like this if I like. However, what I would like to do is to have this repository anytime I want to push, I want these changes that I want to do. So if I have this version that I have, sorry, this version file that I have here, I'm going to make it different. So I'll say atom and maybe make a change to it, something like wrong folder, atom again. Where is my atom? Not this. Where is the atom editor? Oh, here. Yeah. So I can change this version to say number two. So I can change it, number two, save it. And then back in the terminal, I now have a different version, version number is two and my Git status is modified. So I need to send this change to GitHub, which I will do right now or maybe later. But first what I want to do is my local machine has version two, my Git repository is at version one. So this is version one. But now I want to run this first of all version number one in Docker Hub. So I will go to Docker Hub, like here, and I will assign a repository. And I say, you know what, this repository that I have, I want to create another one. And I will call it CI and Demonstrating Continuous Integration. That's a public repository. And I am going to connect it with GitHub. And this is the, my name followed by CI repository that I want to create. And what this will do is you can always push a new image to this repository using the CLI. So that's the CLI method, but I am basically doing it visually right now to help you understand. So I'm going to create this repository. And so this CI name is actually connected to the GitHub location. That's the connection I want to be able to make. And so what I want to be able to do is, first of all, push that name slash CI colon tag name. I think my name choice was different when I built the image. I think I chose the name Cloud Genius. So let me re-erase that and I will build this image again. So I want to revert the Git status first back to version number one. So I'm resetting my Git. And now I am going to cap my version, version is still one now. And now I will build Docker build and give it a tag with my name slash CI and use this. And so now we have a new image called my name slash CI, Docker images. We have this image ready, just created a few minutes ago, it has a new tag. The tag name is my name. So this new name that I want to push to the Docker Hub, I will say Docker push and then this name slash CI, it goes to the Docker Cloud, it's going there. And this is what I'm changing and sending to the Docker Hub right now, which means here I will have in the Docker Hub.com, hub.docker.com, you will have that new login and you will see that new image pushed right there. So this is the image demonstrating continuous integration that you see right now. It's happening. It just pushed. So that is available now in the remote location. This is the repository name. This is also the image available in Docker Cloud and Docker Hub. So we'll now go to Docker Cloud and actually start creating a service using this image. So we'll launch services and we want to name this service something like another CI dash, whatever number is, and you can put a nickname. So CI demo, for example, and you can create, hold on, I need to have this application running all the time. I need to run it in perpetuity. And so that is required. So let me see if I can run that. The thing that we have here, it dies as soon as you don't do anything. So the Docker file says, copy this file and then you have nothing else to do, so it is done. So we have to keep running this thing, doing something. That's what I want to be able to do. I'm basically trying to run something over here. That's what I intend to do. So give me a second, I'm still thinking as to what would make it run perpetually. But even if it is understood that it will die, I want to still be able to demonstrate the concept. It's basically using my image that I have and sending it over to a cloud. That's what I want to be able to do. So I will go to the location where I have to construct, link a provider, which I have done, deploy a node that I don't have any, so I will deploy a first node. And in there, what I want to be able to do is, this is the CI demo machine that I want to create. And in that, I will give it a deploy tag, I will call it CI. And I will create a new machine in DigitalOcean. And in DigitalOcean location, I will select San Francisco number two, and I will select a cheap machine size, and then I will proceed forward. I launch a cluster. So in DigitalOcean, you should see that I currently have nothing, there are no droplets. But through this service that I am using, I will actually create a new droplet. And so I will launch a new cluster. It is going to create a new cluster in DigitalOcean right here, like right now. And by the way, you will see that there is a Slack notification that has come. It says, you know, you created a new cluster. So there is a new machine started in DigitalOcean, San Francisco to that Slack notified me correctly immediately that now you have a new machine there. It also notifies me saying that you have, let me point out that notification to you. It also said, you know what, we have a node update, we have a repository that we sent. We also have, so these are older, 14th of August, so this is old, this is today. Today, we just created a new cluster, that's what we just did. And that's the notification we are receiving back in here. Let me go back to the desktop and see if we can make sure that I demonstrate that part. So it's good to go. The machine is ready. So back in Docker Cloud, we should see that it is telling us it can take 10 minutes, but no, we know it is done, so we can ignore this. And that machine is now deploying. What does that mean? It is actually installing Docker. And that endpoint is going to become a part of this Docker Cloud interface. What is this doing right now? If you look at that machine itself, in here, on that node, is this IP address, is that it is installing Docker. It doesn't come with Docker to begin with, but it is installing Docker. That's what this deploying means. Once the deployment finishes, we can then construct a stack. A stack is basically something that you have running, that you may run applications. So here is an example stack. So I'll open and read the stack itself. So here is a stack file. The stack file is conceptually very similar to Docker Compose file. So here is a stack file. Let's go see what this example stack file does. There are lots of things in here. However, we will create a simpler stack file that will use our own image, and we will create that in form of a new stack. So in this example that you're looking at, where is that? Here, that we ran, we are going to create a new file called stack file that conceptually, structurally looks like this, but not as complex as this. So I'll simplify it to say, you know what, this image will use the lvlnh slash ci. And it has a tag called ci. And we don't have this. We basically don't have anything else. So I'm going to delete all of these guys, basically, just to simplify things. And I have the volumes and working directory, and all of that is gone. So very simple, one image used in this particular stack file. It shows you the stack that I'm building with this image and this tag. And this image basically does nothing except puts this file inside the box, inside that container, using this Docker file syntax. That's what this thing does. So it's very, very simple to understand, at least from this perspective, although it does nothing else beyond. So the process will terminate as a consequence. But if you take this ci demo, if you take this name, ci demo name, and now use this stack file, take it to your Docker cloud, and then create a new stack. And so here is a new stack that we will call ci demo. And paste in this stack file and say, you know what, go and deploy. And as soon as we say, create and deploy, it will actually deploy this particular image on that cloud inside that container. And you will see that the droplet is ready. It is now running Docker, but it doesn't run our image yet. That's what we want to accomplish. We want to send this image over that we have constructed over to that cloud. And so we'll do it in two-step process. First, we'll create the stack. So we'll just say, create a stack. We created a stack. The stack is called ci demo. Now we will see that it notified us again that we have a new cluster ready, finished successfully, and we have created a stack called ci demo. These notifications are coming along in Slack already. Let's go keep forward. So we have this stack. It is not running because we never started. It uses this image called nilesh ci. And I want to now start it. If I start it, it will start the stack and then basically be done running it because the stack does nothing. If you see the stack, it does nothing. After copying, it literally does nothing. So I want to demonstrate this as is without further complicating it. There are lots of things you can do. But I will actually show you that this particular machine, which is here, this one, will actually run. First of all, access it. So I'm going to access it, launch console. So this machine is the console that we are looking at. And I am going to log into this console. We don't have a key pair with that machine because the key pair is controlled by Docker. So we have to inject our own key pair with a separate method. But what I want to be able to do is at least have a method of accessing. So we are using this launch console button for which we need a password. I don't have a key pair. The machine was created by Docker Cloud. So I need to basically reset the machine so I can get a password. And then I can go in there. And so I just reset the root password, which means I will get an email from DigitalOcean that will give me my password for that box. So I can then connect to this IP address that I have here. That's what I intend to really do. So I can see what Docker is doing to my machine as I operate on it. That's my idea. So I'm waiting for the root password to come. It will be in my inbox right here. And that is the password right there. So I will use that password now. And I will use it with this particular machine that is actually rebooting right now. It rebooted because I wanted to get a method to connect to it for which I need to get the root password for which I had to reset it. So now we have that thing running. We also have this terminal available to us that we can use to connect to it. So we will, after this terminal, I would rather not use, I will go to a different terminal, which is where we were. So we were in this terminal. So I will say root, sorry SSH, root at that IP address that I now have, or rather I don't have. So I have to copy that. It is 138.68.56.3. So 138.58.68.58.68.58.3. And then it's asking me for my password, which I have. So I'll give it my password. And then, do I have the IP address correct, 56.3, okay, the IP address is wrong. So break it, 56.3. And then I'll say yes. And then I'll give it my password that I have. And it says, change your password. So I will change it and change it again. It broke the connection, not good. So I'll go again and I'll say, okay, my new password, my assigned password. So it says, give me a password, new password, and do it one more time, please. No, it's breaking the connection. Now, this is crazy. Why are you breaking the connection, sir? Do it again, one last time, otherwise we have a different method to reaching the box. So this is the box. I type in my password and it says, give me a new password. So rather, it says current password. That's why it's breaking the connection. I was making a mistake. So it is saying, give me the current password, please. And so I say, okay, here it is. And then it says, give me a new password. And so I give it. And I will type the new password again, and now it should not break. So now we have a connection to that machine that Docker Cloud has constructed for us. That's great. At least we can see what Docker Cloud is doing with this machine, right? Now we'll put this on this side, make it a little smaller, and understand how Docker Cloud will operate on it. So back to Docker Cloud here. And we have that machine that is not running anything right now. But as you will see, it has Docker installed, and it has a bunch of things already running. These are the management tools that Docker Cloud runs to manage your node. So that's what you have running in these things that are running here. As you can see, all these things are basically Docker Cloud images that helps Docker Cloud manage your bots. If you see this Docker version, it is running 111.112, this version. If you see the Docker es-a, if you see all the processes running right now, all the containers running right now, you will find that there are these many. So all of them are basically Docker related, constructed by Docker Cloud, these four. And these are real works. So some extra, again, by the same Cloud, Docker Cloud. And now we want to run our own image that is available in a stack file that sits like this in there. So there is the stack file that you've seen it. And I want to maybe show it to you one more time before I cancel. This is not the way to see the stack. Stack is available in this location where I need to go and edit the stack. So this is the stack I want to see. And I want to see the stack file. The stack is very simple. It has my image there. And I constructed it for you just now. So now what I want to do is run this particular stack on this droplet here. And then I will start. So as soon as I start, you will see that this is actually pulling that image down. Let's go see what's going on. So we'll say Docker es-a. And you will find that it has a new image that it will pull and then run and then notify me. And I get a notification. It says start action, finish successfully. That basically means that this thing actually ran the Nilesh image somewhere here. So let's go see it now. We will go and make it wider and say Docker images. And you should see that this image, Nilesh slash ci image got picked up and it ran. You will also see that it has exited, that this 30 seconds ago, it exited successfully after doing basically nothing. So if you now see that particular thing running by itself, you saw that this image got picked and it ran and it exited successfully without error because there was nothing to do. Now, if you want to run a different modified approach towards the same image that we have constructed, I want to update that image. I want to also make sure that I go and show you the concept of building automatically. I want this image to be built in the newer version of it. This image was built about whenever it was built. So 20 minutes ago or 10, 15 minutes ago, whenever we did that. What I want to do now is show you the connection between this Docker Cloud and the GitHub location. So that GitHub location is somewhere here, back to my name, followed by ci. So that's a public repository we now have, and I want to make sure that this is connected with Docker Cloud. And so I will look at that repository and see what repositories do we have. So we have this repository already. We want to make sure that we... So what I want to be able to do is make sure that I update my build process. So I will exactly look at that. First of all, let me quickly check one note that I have. So I will quickly check that note that I have here, automated build. This is what I need to enable. And so Docker Cloud build is currently free while in beta. So that is what I want to make sure that automatically build from GitHub. This is how I need to connect. And so for that, I need to go back to my repository and make sure that the building is actually automated. So this aspect that I want to automatically build is automatically builds from GitHub. This repository is enabled automatically built from a GitHub repository. If you want to edit the repository, you can make that changes if you like. But you see that this demonstrating CI, build location, and the actual pickup, it says further that if you go look at automated, auto build triggers a new build with every git push to your source code repository, you're going to learn more about it. And it shows how it is going to work, so you can read that. And the idea essentially is that I want to now make changes to my source code and watch this thing build. And I will show that this will build in this location. Where is the Docker Hub location? Here. I think this is where the build happens. So we have to see the webhook. Let me see what happens if I actually make changes to my code set. So we have the code here that you see in the code in this location. I want to make a version change. So cat, atom, and make a change to the version. So the version is now version number two. And I'll save it. And all I will do instead of building it locally is I will actually do a git add, git add, and then git commit, and then git push. And as soon as I push, it goes to GitHub first thing. So let's go see the GitHub thing first. And you will see that this GitHub location has a new version now, which is saying version two. And you could also see automated builds happening as a consequence of this push in your Docker Cloud somewhere. Where is that? Hold on. Not here. Not here. Docker Cloud. Where is Docker Hub? Here. Sitting. No, that is Hub. Yeah, so this is this is Hub. And so this should actually get a new push, the build should pick up. So this is the location where we should see a new build coming up. This was 20 minutes ago, the latest build. If you see the settings, and Webhooks and collaborators, where is the link that I want to see? Hold on. It should be somewhere here. Stacks, repositories, this should be building right now. It says Docker tag, local image tag. You can always push new image to this repository using CLI, but I want it to be automated and it tells me that it is automated. So why are you not showing how do I find and see? So it says 26 minutes ago, so this is the older image. But I want to see the newer image built correctly. I'm missing something here that actually shows you where it builds. So that's what I'm trying to find out where exactly it is. It is not this, not this, we're not getting notifications about that. We should get notifications about a new build popping up in Slack already as soon as this thing happens. But apparently I'm missing some connection somewhere along the way in between this automated build and that repository that we have in there. So the Git repository should automatically push. Let me run these guys, Docker tag, local image, let's push local image tag name to tag names, Docker tag, Docker push. No, this is not the idea. Now I will come up with something that I'm missing as to where exactly this automation is visible. You can see it happening. I'm not able to quickly locate it for some reason because I haven't done it in a while already. Let's see. What about 22 minutes ago, there were 11 holes loading, hold on, where are the automated builds? So you want to see the, I want to see the build process. So it should automate. It should happen already. It may have happened already, by the way, by now, but I want to see it. That's what I was trying to see. Maybe instead of wasting time on this, I will look it up and then give you specific links on how do you connect these guys. Yeah. My UI is a little different than yours. I think you are still in the classic UI of cloud, but if you go to the repo and under builds tab. Okay. So let me switch to the new UI and go to repositories first one, then builds tab, you're on general tab. So here, I think it's configured. Yeah. I think it is, I've done it by the way, I have done this many times apparently and I'm not able to show the, they must have changed something as to where they show. You can actually see the build happening. That's what I was trying to find out, but not right now, apparently. So we'll do this later. But in the bottom line summary is basically this. You have a Git repository that is your laptop. You send it to GitHub and from GitHub, it will automatically pick up changes in Docker Hub. Docker Hub will build image for you and Docker Cloud will take that image, use it for you, use the new image and automatically delete the old one is gone. The new one is picked up. It happens very smoothly without a problem and it happens like seamlessly. So all you need to focus on is here. That's it. And push. Make changes. Make push. That's it. That's all you do. Everything else is, it goes live by itself and you can scale it here comfortably. And Docker Cloud will scale it for you simply by looking at the stack that you have created. So here I have this stack, CI demo, and I will start it again. And then I get a notification that the stack ran. So here we have Slack chat and the stack says demo started, demo finished. That's pretty much it did. Well, that's what you are able to operate on with the idea once you have this running like that. So I will maybe give you one more example, but I will let you play with this concept of continuous integration like you're looking at right now. And I think that's the thing that will help you in building your Rails application that we were talking about the other day, Rails app. Yes. Put that in Docker. Put that in CI through Docker Cloud. And then that is sweet. It is simple, really simple. And also there are like this Docker CI, Docker Cloud CI, there are other variations to Jenkins. Like for example, does it, you will have a bunch of other services available out there in the industry. Some of them are open source. Some of them are not open source. They all help you accomplish this goal and basically operate pretty much like this. You have a Git repository, you make changes, you push, that's it. You have not, you don't have to worry about anything else. All of this, somebody else handles it. This Jenkins-like tool will handle it or Docker Cloud continuous integration will handle it. That's the concept. I will show you one other demo, maybe for a live application, I have this one application sitting somewhere. I've used it four months ago, but that I haven't been touching it. Last push was four months ago. So I can go back and see if I can show you that, that's a live demo and it is very visual. So you can actually relate to it much more easier than this terminal business that we have. This was not really visual to understand. Okay. So let's, we will revisit this, revisit CI tools later, right? Let's see how many people are now online. Is it still two or somebody else? Yeah. Dennis is also there. Oh, hi Dennis. Hello. And so I guess people are busy today for some reason. And so Dennis, we were talking about continuous integration with Docker Cloud. And so just briefly tried that there is a thing that I wanted to demonstrate that I could not find properly and quickly. So we'll get back to that one later. But right now, what I want to do is switch back to the topic that we were covering the other day, which is architecture, architecture here. And I also see people on Marjorie is there, but she's not online. So we should say hi to her. And she is online. So back in here, these are some of the diagrams that we started to look at. And I want to make sure that we cover some of the business usage scenarios of a variety of different architectural designs laid out here as reference architectures. We covered only one of them by now, which was this one, which is media sharing, if you recall that for Netflix. But we haven't actually studied the other ideas in terms of what a variety of businesses actually use Cloud for in what types of architectures people put together, and what are considered best practices. So here is actually a collection of best practices. And that's what I want to go over, not spend too much time on these, but at least want to make sure that we cover a variety of different business use cases. So this first one is web application hosting, which is probably very, very simple to understand. It's not that complex. The concept essentially is that you have some application you want to run and that you want to be able to serve people with your application. So in a classic, simple sense, you require some things like a web server, an application server. So these are the application servers here and also here. And typically, the best practice essentially boils down to that you put these guys in two different availability zones. So this is the A, availability zone A, and here is B. And so you typically put everything in two distinctly different geographically separate. And also, conceptually, they do in such a way that if there are one area is prone to earthquakes, then they will find an area which is less prone to earthquakes. But if one area is prone to flooding, then they will have another area prone to not flooding. So they've tried to divide their availability zones in separate geographical regions within the same, say, what Amazon uses, the word is region, I think. So I should use that word, what Amazon uses in the context of what we have. And so let me just make sure that I use the right terminology. I think Amazon uses the region and then availability zones. So the concept of this thing are called regions. So we have in Northern California, we will have one A, one B, and one C availability zones. So you can see those. If you start to start a machine, you can see them in somewhere here. And you can see that, you know what, select in one A or one C. You can select in any of these two. We seem to have two subnets in there, one A, one is one A, one is one C. We can create a new subnet if you like, and that will become the third geographically separate. And you can put this new subnet in your one B, which we don't have. We have in one A and one C, but not in one B. So you can create that if you like. And that's how you decide where to put your machines at a given time. And so that boils down to the idea, which is you have this A region, they call it availability zone, and this is the B availability zone. So you will put things in two separate distinctly different locations within the same region, like the Ohio region or the Oregon region or the Northern California region or Virginia region, but have them in distinctly different availability zones like this, just to improve fault tolerance. And so you have web servers here as well as web servers here. You have application servers here and application servers here, database servers here, this M database server, which is master, and another one, which is a slave database server. So the idea is to synchronously replicate the databases on the backend between this RDS database, which is the master, and here is your standby, which is also an RDS deployment. And these are services available from this example company from Amazon. If you don't want to use a database service like RDS here and here, you always have this solution that you can use in open source where you can implement something like this box and something like this box, very, very simply in say, Galera and Maria. So Galera is a cluster technology that basically lets you run exactly what these guys are doing, for example. So here is the software, MariaDB Cluster, Galera Cluster. And the idea, this thing is open source, by the way. So it basically gives you something very similar here and here, master and standby. So you put one box here in the A availability zone, another box in the B availability zone, and run MariaDB Galera Cluster among the two. And so you will get this solution similar to what you have here and here. That's the MariaDB-based solution. You can similarly implement other solutions for other databases that you may prefer, such as Postgres or MySQL. All of them have this method of replication of some sorts that allows you to replicate content, the database content from this location to this location within different availability zones. And that's the database layer at the very, very back end, at the far end. And prior to that, you will have application servers, which will run your applications that will actually request data from the database server and then render that over to a central point, which is here in this example. They are using two different elastic load balancers. One is here and the other one is here. The scenario behind this is different from this. This load balancing service is internal load balancing. It is load balancing among the internal application servers that you have inside your virtual private cloud that spans zone A and zone B. And this load balancing is a service available from Amazon. However, you don't need to use their service. You can always use this, which is this open source application called HA Proxy. And it has beautiful, basically, I suspect, I don't know for sure, but I suspect these technologies that they use here and here are actually HA Proxy. They're based on this product is what they have running here and here. The elastic load balancing service that you can find in Amazon Cloud. So if you go to console, you will find elastic load balancing service right somewhere here, I think in EC2. And so on the right side, the load balancer solution available, you can pick and choose to create a load balancer. And that's this load balancer solution. You can just quickly proceed ahead and then decide how many are internal or external. What do you want? So in the example, you are looking at both. So here, both two choices are available. In the picture, you have this is your external public facing load balancer, whereas this is your internal load balancer. And that's what you are looking at here. When you configure a load balancer, you can say internal facing or internal. One of those two you can get. And then this idea behind putting a load balancer in between every layer is to basically load balancer and select appropriate number of application servers that your customer may need. This guy, they may hit your services hard. And if your services are getting hard hit, the idea essentially is that you want to scale your web servers. If the number of web servers you have on this yellow band is auto scaling, it will automatically scale to a number that meets the need of this customer, the end user. And if there is not much need of this web server services, then it will auto shrink or auto make itself bigger. So basically grow and shrink automatically based on the need for web server application. So the web server servers, these boxes, how many you need will be decided based on measurement of performance for this guy. So for every step of the way, for everything that you run, so all of these items that you have in a given infrastructure, in a given layout for an architecture, you will have to measure performance of each and every item that you have across the board. And based on how much of a performance drag that you are running into, these application servers might actually get bogged down based on how much load you're being put, the users are putting on these servers. And so if they are getting bogged down, the performance measurements will indicate that they are actually slow. And if that's the case, this auto scaling needs to look at that performance metric and automatically scale and grow to spread the load among these other boxes. So you have more boxes that will participate as application servers and automatically scale this yellow band for EC2 inside your VPC. So it will add more boxes like this, add more application servers on both A and B availability zone, primarily to make sure that the performance overall on each of these boxes, the CPU consumption, the memory consumption, the disk IO per second consumption, all these metrics, you need to measure these continuously on every box along the way. And based on that data that we collect, we have to automatically scale these guys to grow or to shrink. For example, if the performance goes something like we are collecting statistics on CPU consumption or memory consumption or disk IO, any of these characteristics. And if you find that the CPU consumption is actually hitting 80% continuously, that's a good sign that you are reasonably employed, or you're employing all these CPUs nicely. However, if you see that this CPU consumption is going higher and reaching say 90, 95, that's a problem area, you know that you are actually getting hard hit. And that's a signal to the load balancing algorithm or the auto scaling algorithm will look at automatically growing the number of your servers or make this number of CPUs available increase automatically grow. And when you grow the number of CPUs, you need to communicate that back to the load balancer that I now have more. And this communication actually happens automatically. Something like you will see in here in this illustration that I'm pulling up. So you have this scenario where you see that these boxes are not there anymore yet. So these two boxes that you have, you notice that the performance of the CPU here is 80%. And here is also 80%, and that's great. But then at some point, these customers come along, more of them come, and then your CPU consumption grows higher. When it grows higher, you need to bring more boxes. So this was 90, and this also was 90, 92, and that becomes a problem spot. You want to not go beyond certain threshold. And so you basically define these thresholds inside in your auto scaling algorithm, which is very easy to actually implement. It is already done for you in places like Amazon. So you have to decide whether at what threshold you will actually pick and add one more box. And you can do that, and you put that in multiple availability zones, like one put in the A zone, the other one goes in the B zone. And now you have four boxes. So from a performance of, say, 85, 80 or 85, the moment you add more boxes, it will bring it down. The CPU consumption will go down automatically, and it will go to something like 50, or maybe 40, whatever that number may be. 45, 45. Whatever percent will spread the law, basically, on the workload that gets allocated to these boxes. However, you need to have a systematic method of communicating back to the load balancer. Did you know what? I just added these two new boxes for you. There were only two to begin with, but now you have four. So you have four places to send your work for these guys. They will come. They will visit. They all visit this point. This is the load balancer. And this load balancer needs to be aware of the fact that you now have added more boxes. At some point, the work that these guys are asking you to do will reduce. And in that case, you will find that the performance characteristic of these boxes are showing only 10% CPU. That's a wasteful allocation of resource. So you need to find a method in which to account for your auto-scaling, auto-shrinking method, to account for the fact that any time performance numbers of CPU consumption go below a certain threshold, that you need to start killing your boxes automatically. So you need to come back here and say, you know what, let me kill you and you. And so you also need to communicate to this guy that, you know what, there used to be this box, but it's not there anymore. So just remember that you have only this and this. And those boxes are gone. So I just want you to know. And that information has to be passed to the load balancers, that the load balancers can then operate accordingly. And then send workloads only to this and this. And then the workload of these boxes will become 20% each. And that's great. And you can continuously track, continuously monitor all these characteristics that you're looking at here. So how much CPU? How much memory? How much disk IO? How much this and how much that? And you can come up with those metrics on your own and measure. And based on measurement, you will auto-grow, auto-shrink. And that's the auto-scaling algorithm that you have to accommodate and actually set thresholds. So basically, you set thresholds for going up, threshold for grow, and then some time elapsed to measure. And then after some time elapsation, say five minutes, 10 minutes, then you set another go low threshold. So for the last five minutes or 10 minutes, I'm not seeing any performance consumption more than, say, 10%. It's all less than 10%, I need to start killing. So auto-shrink policy will kick in. And those thresholds, we have to come up with for every application. You have to decide what are those metrics that you need to measure and what will actually constitute a grow policy or a shrink policy to kick in. And that's what it will take shape in this example that you're looking at. So if you take that back to here, this load balancer is basically keeping track of and also balancing the load on these boxes, the application servers here and here in two different zones A and B. And that's the auto-scaling concept going along, focusing primarily on the application server, these guys. They in turn will connect with the RDS. And they have not shown you a load balancing for RDS because it's a managed service. It automatically does it for you. So you don't even have to worry about these things. In fact, you will see that there is no link shown here, as you can probably notice. There is no link that goes in between. Let me point it out for you. So you see this red line here? No, it's the red line. Draw. Now I can draw. So you see the red line here? This line. I cannot draw yet. You see the red line here? But you don't see this red line here. There is no such thing. The reason it is not there is because it is a managed service. You don't need to worry about how the database scales. This connection happens exclusively through this point. And if that point itself gets broken, this service will take care of whatever you're trying to accomplish. So this RDS service will automatically handle failover to another box, another standby box, depending on how this box is performing. And it automatically makes them, adjusts them, and that's a paid service that is available to us in your, let me erase this. I think this is clumsy, but let me get rid of it. So that RDS service is a paid for service, by the way. It is a storage solution for databases that is available and it automatically scales. So you can start a MariaDB, select and say, you know, for multiple availability zone deployment, give me a high availability zone solution. So you can just select and say that I want to have MariaDB provisioned in multiple availability zones. It will do it for you. That's the idea behind this and this. That gets deployed. So that's the reason why you don't have this need for a separate auto load balancing because that part is taken care of for us in RDS. It's a service. It's a paid service. They automatically do it for us if you use their services. Back in here, you will see that very, very similar to this, this idea of elastic load balancer, which is basically scaling up web servers for you up and down. I should say grow and shrink horizontally out and in these web servers will handle traffic from these users that get hit at this endpoint where the load balancer is actually distributing the load among these boxes, web server boxes between A and B availability zones. The conceptual distinction between a web server and the application server is something that I would like you to understand if you do not know this. So I would like to explain that because there's a subtle difference in how these two are different. The concept essentially is that if you look at, let's say this page, this page itself has this data that I have written and it has this image that you can see that looks like this and it has a bunch of other components. So if you take any site, any web hosting example, so yeah, that's what you're looking at is a web hosting example and you will find that a web hosting application will have some elements that are static in nature, like this picture, for example, it's a very static picture. It doesn't change unless the logo itself changes. So it's not really a dynamic object that changes itself every second or every minute or something like that. However, these statistics that you will see here on top, they change quite often. Similarly, the data of the user logging in, user logging out, that is also changes quite rapidly. So you have to handle the fact that this entire web page that you're looking at anywhere you go is handled by separate components. A typical scenario that you have a site that a user visits, the user receives some content that is static in nature and other content that is dynamic in nature. The content that is static is mostly things like text or JPEG or PNG files or video files or sometimes JavaScript or sometimes CSS elements and HTML itself. That's the concept that doesn't change too much. HTML frame itself, I should say. However, a lot of things are dynamic in nature and those things are actually stored all the way to the back in the database. To extract information outside the database and make something useful, you need some application. That's what is an app server, typically. And to serve these things like pictures and videos and JavaScript and CSS and HTML frame, you don't want to bother this guy. It has a specific purpose. The purpose is to generate content dynamically through database sources. And so that is the purpose behind this app server, so you want to dedicate that box to that purpose. And for other things, you don't bog down the app server. So how do you serve? That's where a web server comes in. And a web server typically is NGINX. Very commonly, that's what people use to serve static content like pictures and videos and JavaScript and CSS and HTML frames and everything else that doesn't change. That is those portions are served by the web server. Anything that dynamically changes on the fly that needs interaction with the database is typically an application and is typically driven by an app server. That's what you're looking at in this example. So the application server is actually interfacing with the database and is then producing the dynamic content, which in turn is given back to the web server to couple with other not so dynamic content and is then rendered to the user. Some parts of it are offloaded, like pictures are typically offloaded to a bucket like here and it's where the static content typically sits. It reduces the workload on your web server even further by offloading that item of a picture stored in a bucket that gets served through a content delivery network like CloudFront. And this is basically outsourcing the work that these guys are doing so you don't have to do it. I've shown you this example where the picture that I have here is actually outsourced. So for example, let's go look at the copy image address and open up in notepad, item and see it. You will see that this picture itself is in a content delivery network in a folder called WPContentsUploads and that type of picture. But this location is not the website. The website is not that URL, but different URL like this. That's the URL for the site. And the item is actually coming down to you through a separate location, which is a content delivery network, CDN. In our example, here is the content delivery network, which is CloudFront, which actually stores the item in a bucket. And so this is where it is picking up the picture and showing it to the user through CloudFront. It is not even bothering your web servers, forget application and database. That doesn't need to worry about it. The pictures directly come straight from the bucket through the CloudFront network to the user. That's the shortest, sweetest, and also simplest method of delivery is just put in the bucket. It goes through CDN and to the user. You don't have to worry about servers at all. So all these guys don't bother with pictures and don't bother with much of the static content. That's the method that these web servers are. But then you don't have such a bucket available, or you don't want to use a bucket. In that case, you can put your static content and serve it through these web servers. That also works. That's the idea behind web servers and different and distinct from the application servers, but they work with dynamic content. These guys work with static content mostly, and they're also load balanced. And you saw the buckets and CloudFront, and that's how the entire package is constructed and given to the user. The user discovers your services through some kind of resolution, DNS resolution that happens through CloudRoute 53. We have discussed some of this concept already in a variety of examples that we have done. And so that's a typical web hosting scenario. The challenging part of running this kind of a service in running your applications, like say here and here and wherever you are, is to be able to run these things in a fashion that doesn't tie you to a particular cloud. And that is something that is a thing that you want to basically unshackle yourself from a particular cloud. So for example, this site that you are looking at right now, this site should not depend on one particular cloud for any reason, and that's how I have constructed it. So in effect, what I would like to demonstrate to you is how you take this entire site and migrate it to another location through complete hands-free automation. That's what I would like to show you. That basically relates to this example of a web application hosting. I'm hosting the CloudGenius site as an application for you as a customer. And I would like to demonstrate this idea where you can take this entire stack. I have all these layers, by the way, that I use. And I will take them to a different cloud on demand, pretty much. And I want to show you how to do that. And we can do that during the break. So during the break, we will just switch our cloud providers on the fly live. So I want to take a short break. And while I am taking that break, you will see this operation will finish up. And I will initiate that process in just a sec before I head out to take a break. So let me just quickly check my source code for migration and make sure that it is all intact. And then I will kick off the process. You will watch it happening and you will see that we will migrate live CloudGenius site from one location to the other without much trouble. And that's what is going to happen as soon as I kick off that process of moving this CloudGenius site from wherever it is to a different location. And I choose DigitalOcean right now. That is what I intend to do. So just making one more edit, making sure that my course, my source code is in a good shape for a live demonstration. And so I will bring it back to how it was and do one more change apparently somewhere here. Another change is good. And so back in the source code, I have only one change. So I'm ready to play. What I will do now as I take a break is kick off a new build to a different cloud. And that's what I intend to do. So I'm basically taking this site that I have, this one, to DigitalOcean right now as I take a break. So during the break, you should find that right now we have this Docker cloud. I want to destroy that. And so that my account is completely empty. So I've just killed it. I will create a droplet or rather I will completely automate the whole thing without even touching this site. So I'm not doing anything on this site at all. All I'm showing you is that there is nothing. And then I will kick off my migration process. And so here it begins. I will make the font a little bit bigger and then initiate Ansible deployment of a playbook to build and construct a new site as I take a break. And I am going to note time, it is 6.27 and I kick off and I did. So this should build and take the entire live site to a new location inside DigitalOcean. You should see that it is already doing it. So it is creating a droplet right now inside DigitalOcean as we speak. So we should see a new droplet popping up right here. And there it is. So as soon as this droplet pops up, it will populate it with the content, it will go to the current site, extract. I have a backup copy also. So it basically extracts everything from there and loads it up in this new machine and tell me that it is done. And so while it goes, I will take a short break and right now it is 6.28. So we'll resume at say 6.35 or so, but we'll keep this running on the side. So I'm just taking a short break, I'll be back. You will watch what it is doing, I'll make it a little bigger so you can understand and read some plays that happen in using Ansible and Docker. So you will see these things happening. It's not super easy to understand, but I will walk you through as we go along. But you will see that it is basically doing a couple of things that are probably obvious to you. It's installing Python or something failed, let's see what failed. Okay, I think something went wrong. So I'm going to break it. I forgot to make a change in the host file here. Something went wrong. So I'll do it again. Let's see if the site is still active. I will delete this one and destroy, go bye bye and make sure that the site is still up and running. Yes, it is. So the original site is still active. And I think I did not catch that mistake that I had. So I will catch it. I just corrected that and I will make sure that I fix it, which I did. So I'll resume that operation one more time. So now that digital ocean should be empty by now again, yes, and keep it off one more time. Let it run. I think this should not error and I'll be back in just a minute as it goes. And this actually is a method of live migrating a current site to a different location, a different cloud where it's not running right now. So I'm taking the whole thing to a different cloud as a live transaction. So live migration of sites, live migration of active running containers to a different location. And at the end of this, I will test it and then actually change the DNS. When I change the DNS, I can actually kill the old ones that may exist. That's what I will do once I am happy with the new deployment. I can kill the old one or I decide to not kill either one of those is okay. And we are doing this on a live site by the way. That's the scary part, but yeah, I'm okay with that. Live site with you guys watching, that's even more scary part, but we'll see. Thank you. you you you you you you you if you were watching during the break you probably saw this that the bots rebooted and this script waited for it to reboot and connected it again and then it is started starting to push some configuration files pushing Docker compose running the stack right now that should bring it up automatically in that new location so basically I'm running Docker compose on the remote location on this box somewhere here this box and once this compose and all the new content it will take content from here in this site and actually show it to us in that new site the exact same thing that we have that takes a database dump I take all the files down and we'll practically move everything here from the live site to another site another another newly constructed location that's what the stack is getting built right now you can see these things if I decide to go copy that IP address and open it in a new stack in a new SSH terminal you should observe what is going on right there you can see that this this machine this box that I just created through automation is actually running Docker and it's doing a bunch of things that this stack is basically getting built right now right there so while this happens I will continue our discussion which is a different architectural layout so we'll pick another one we discussed this briefly web application hosting and I think you understand and if you do not or if you have a question on any parts of it please speak away now later anything is fine but I want to switch to a different topic to a different scenario which is about content and media serving and this is a conceptually somewhat similar to what we did in Netflix somewhat not all of it but this is mostly about one-way traffic it's serving the other part which is to receive content from Hollywood that piece is not there in this illustration but essentially in this illustration what you have is the idea that you are serving content to people so you to solve content you need to have content and you can have content in buckets in your s3 buckets like here that you can serve directly to the end user either through the bucket or through the cloud for distribution content distribution that is delivered to the customer through cloud front so you can do that without actually putting up any new servers just put the item in the bucket and render that directly to the car to the user or go through a cloud front cache or CDN cache on the other side you can have custom origins did I did I discuss the word custom origin never before in with you guys do you know what custom origin is or not did we do that I don't think with it so let me just briefly explain what a custom origin is so it has a the idea of custom origin basically connects with this service called cloud front so the concept of custom origin boils down to the simple idea that you have some kind of a content delivery network that gives content to the user the CDN is like a cache where it looks at a bucket and you have some item here this item I and so if the CDN will make this item I available to the user on a global scale so globally available so multiple locations multi location and you can see these multiple locations in another terminal and we will say show me the locations that I have for this cloud front distribution that I am running right now so show me it so shows me these eight locations I have it shared out through cloud front so multiple locations worldwide that's what CDN does and the origin location for this item called I is actually in the bucket it's this item that's the origin so what is a custom origin a custom origin is basically anything but not a bucket that's a custom origin cloud front is designed with buckets in mind but it doesn't serve just bucket content it can serve custom origin content it means absolutely anything other than a bucket which means you know if you have a digital ocean droplet running some service you can put cloud front layer around it and serve it to CDN that's a custom origin just an example if you have Google Cloud machine running in the Google Cloud and you want to use cloud front that is also possible you can just use this as a custom origin that's what this idea essentially is so if you go back in the picture these custom origins are basically shown as a private server you might have in your garage or in the office becomes an origin a custom origin where you have content will be served to the customer through cloud front you can do it through private server you can do through EC2 machine that you might have that also becomes a custom origin or you can have some live content like you are streaming some live content like your Facebook live YouTube live your web camera you might have some sources you can use those as your custom origin also and for that there is a different method that you need to apply cloud front the cloud front usage that we have shown and discussed up until now is basically serving static content whereas a live stream source is not static it is dynamically changing streaming video streaming movie so you might have a webcam in your hand that you want to stream you can use the same service cloud front in its live movie mode different from static content mode and still stream live video and you can have that served on your domain live movie so this live stream video that you have can go all the way to the end user on your domain that was a bad link it goes to you it has no meaning to it but that's just a dummy link that they have given here that shows that this is where you can drive your traffic to your users can actually stream the content that you may be sharing live right here so an example of that would be something like here I'm going to pull up something to show you that idea so here on November 7th at 12 o'clock I will actually open up my live stream in which people can watch that live stream pretty much like this and in 10 days it will open right in that location that's the idea that's what we are looking at in this example where we have a live stream source which is my webcam I will open it up and not use easy to but YouTube services pretty similar like this and stream live content worldwide so same same thing same concept so basically this illustration is of serving content in serving media through content delivery network in case of Amazon it is called cloud front every cloud company has their own content delivery network and all of them operate pretty much the same way that you can decide whether you want to stream your web camera or your high end broadcast camera or other types of devices that stream video like your iPhone for example they can stream videos and you can take them all the way through cloud front or through some other content delivery network to the end user and people can watch your iPhone if you are set up like that through a CDN without any issues without any hiccups without burdening the bandwidth on your iPhone that's the idea core idea is your camera your iPhone may not have the bandwidth to serve the end users globally but the moment you put some kind of a cloud front like content delivery network in place that your camera can go live worldwide and users worldwide can see what you're streaming live and that's what this capability that content delivery networks provide is the core idea behind what they are describing in this section media services media streaming services and you see Microsoft Azure has this similar thing as your media services they have a service that you can basically stream pretty much similar and the idea is you stream your video and they will serve it for you globally that's the concept every cloud provider will have such a thing that you can take your live stream video and send or you can take your content that you might have in a bucket in your s3 or blob store in Azure or Google content store or any other about any other place to store stuff and serve the content like PDF file pictures movie files those are static content if you don't have them in a bucket like this you can maybe have it in a private server in your garage or in your PC2 location or Azure cloud or Google cloud or digital ocean cloud those are custom origins different from buckets and you can serve them also through cloudfront like distribution like CDN for example and so Akamai if you know of a company is the oldest content delivery network that people know probably you've heard of this Akamai and this is one of the oldest established names and then we have these days a ton of them how many CDNs we have we have a huge collection of those so let's see content delivery network who are the providers and you know many of these guys already by name so it is a long list of companies that are in the business of content delivery network that include Akamai and cloudfront and this and that bunch of those lots of those so basically all of them do the same thing and the idea is to deliver content it doesn't change by itself as well as movies that are streaming live both can be served in this model that's the model of delivery to the end-user where you can serve content and streaming content streaming media both that's this idea let us see that deployment finished and there was some issue with the upgrade pivot portion but apparently that particular thing may have failed but the site should be live I'm pretty sure the site is live so let's go test it so I'm going to quit this Chrome box thing and quit this and quit this and then go to get the IP address so the IP address should be available from digital ocean and let's go find out what was the machine IP address that we obtained and so the IP address is right here we copy it then we go here to the terminal backup and then I will edit my zone file or other my host file so here and this is like a local test that I want to be able to do so I'm going to basically modify locally hack my machine so that it will look at this new deployment that I have just constructed and so I'm pointing all my names domain names including this new domain that I have which is cloud cloud cloud that's the name so in in that example what I just did I think I made a mistake so let me quit that mistake if you quit and do it again what I just did was copy this IP address and then put it in this location to remove the comment and now I have my domains pointing to this newly constructed digital ocean IP address which is nice so I can then ping and I should see that it is going to the new IP address which is exactly what I wanted right I wanted to go to 103 that's what it's going 103 bingo let's go test now the test will work only on my machine not anybody else because the global DNS is not changed yet however I will now open this site and it should go to the new droplet and automatically show me the right site exactly how it is and there it comes you will find that this is a live functional site except only I can access it right now because the IP address DNS IP address doesn't actually change until I make a modification to the DNS which I haven't done because I don't want to unless I am feeling comfortable with this setup on this IP address but I once I feel comfortable with this deployment I can maybe go check my dashboard and see whether the programs show up correctly and all that looks like they don't or maybe they do but I need to make sure that it is actually doing the right way and if that's the case am I able to buy stuff as a customer probably I can and things like that I will examine and see if it is actually working looks like it is working if that's the case I will probably test it by the way and then go to cloud player and modify DNS once I modify my DNS I should be able to see in that location which is here login and change my DNS and go to this location modify the DNS the IP address for the main site and modify this location to the new IP address like this should get the number from digital ocean like copy and then paste once I do this everybody will have the access to the new site and I can then retire the old site if I like I haven't tested it thoroughly so I'm not in a mood to change it so I'm not changing anything I go back and then leave it like the way it was so I will leave it the way it was and not change actually but I can and you just saw me migrating live after having done that part I want to kill this box so I destroy it and here it goes away so I can basically kill the new newly constructed box because I haven't had the time that I really want to make sure that I feel comfortable and this is a live site and I don't want to make a mess that's why I killed it but the concept that I basically demonstrated to you is that you can actually take live functioning sites and migrate them on the fly on your whim whenever you feel like whenever you feel like I don't like this cloud anymore I want to go to a different cloud fine go that lets you let you have the ability to go if you want to I don't necessarily want to go the thing is running fine like I will find that if I open the site right now it will not work because my DNS points to the broken box so if you open Firefox it will fail because the box is dead and this doesn't open anything right now so if I try to refresh it will not open because my zone this file, host file is incorrectly pointing to a broken box I need to hide it and then close and now I ping the cloudgme site I should see that it goes to the correct location and now I can go back to Firefox and open like everybody else so here we go it should now bring it up and so it's opening the site right now so I'll refresh and it will bring up the current site that everybody else in the world looks at right now that should pop up any moment right in this window in Firefox okay and let's also check it in the other location the refresh that one should show up yes and another browser and the site refresh I think it is working yeah so this should work which should be working already for all of you except not Firefox for me for some reason which is fine so you saw live migrations you also saw that we discussed this which is content and media serving concepts it will keep following through quickly and I want to make sure that I cover these ideas these concepts in terms of variety of use cases of how cloud can be used for back processing in the next example I'll open up this one before that I want to change over do you have any questions from these guys if so please ask away if this is too trivial for you please tell me that I will stop and do something else but if you want me to cover these concepts please say so and if you don't want me to cover please say that also because I don't want to bore you by the way so we have a lot more things that are interest to you we'll cover that instead instead of walking you through these conceptual diagrams if that's the case please tell me so that you don't want to go through these things because they are trivial or simple or too easy to understand so I want to hear from you I did not hear I think we should go through these things as well okay okay so we will not spend too much time but at least I'm gonna cover every conceptual idea behind these illustrations because they are very nicely done for a variety of use cases so here is an example of what is generally known as batch processing and a batch processing is basically something that I would like to draw a picture to give you an example so you want to get driver license from the Department of Licensing so you go to that location and you are first time getting a license so you have to be in person so the person has to go there and stand in line and then the officer here this officer number one will actually meet you look at your paperwork ask you question take make you take an exam or things like that and then give you something do they give you the license right away the actual license right away no they don't they probably give you some kind of a temporary paper temporary paper to say you know what yeah you're done that's one example another simpler example like this is that you go to the Social Security Administration and you apply for a Social Security card the officer will give you some kind of a temporary paper right away and don't actually give you the Social Security card that looks kind of whatever it does look like so they're not able to give you this thing right away so what they are doing as a consequence is that they give you a temporary piece of paper to show that you have what it takes or whatever you need but the actual paper that the real driver license or the real Social Security card you will receive in the mail a couple of weeks down the road and that is a classic example of batch processing where they are what they are really doing is that they are creating a queue for the back end back office to work on the back office will be basically populated by less capable people than this officer and all they will do is look at this queue where it says here is this item that I need to create a Social Security card for Ram and another one Social Security card for Mukesh like that and so the the back office basically is probably either a person of slightly lower capability than this person or a machine that looks at the queue and is operating in batch mode it will do these things in next three hours maybe next three days and then it might decide to go on vacation or it might decide to you know stop working for you or something like that and maybe go away to do something else in their life and that's okay because it's a batch job it is like not critical not time-critical this thing with the person is waiting for you in line you need to give them something right away it is time-critical and so the things that are time-critical you need to use you know machines on demand machines on contract but the things that are not time-critical you can use spot machines for worker nodes you've seen that idea before and that's the concept but in any kind of a batch processing situation where the thing is not critical from a time perspective that this person is waiting for you to give them some piece of paper at least right now so they can feel happy about coming to visit you to get a driver license or social security card but then they know that up about four weeks down the road somebody in the back office will create that card and give it to you mail it to you and if it takes five weeks instead of four it's not going to hurt you it doesn't you know it might take eight weeks who cares practically you don't that's the idea behind batch processing so anytime you have this concept of a queue of things to do that you put them in a batch you handle these many things right now in a batch you do them if you're done that's great if you fail then you put them item back in the queue and you continue the next batch so basically to take it from the top and take a next batch and then attempt to do that exercise and so if it finishes that nice if it doesn't finish then you put it back in the queue that's the idea behind batch processing now in this picture what they're really trying to communicate is that there are a bunch of queues and so you can use the Amazon's queue service to manage some kind of a queue that keeps track of things to do for somebody like here so there is a job that comes in like create a social security card create a driver license and then the user of this batch processor batch process unit is basically creating new jobs that needs to happen these jobs will get assigned in a queue and stand there until you have a large number of these items in the queue at which time you will increase the number of workers that you might have in an auto scaling fashion based on the size of the queue the auto scaling auto grows if the queue becomes longer the work gets piled up you scale the number of machines you have in your workers and by the way you will notice that they're not bothering here to show you the concept of high availability because it's not really relevant you don't need to worry too much about high availability in a batch situation because if something goes wrong you can just put the item back in the queue and do it again what's the big deal it's not time critical the end user that is actually end user is not worried about that's the concept so the idea behind these guys working on the queue is to do some work so in in example of this driver license I mentioned and you know social security guards that I mentioned that's great but what's the the actual example that I might want to give you in the context of a real application so here is a user that visits a site and there the user fills out a login name and so basically he's trying to register so there's a login name and a password and email address and bunch of other information is requested okay and this login and email needs to be validated so this validation of email typically happens through some machine sending an email back to the user so go check your email and click and if you click then this email becomes valid that's the typical protocol that people will find that the email validation happens so this email has to generate and has to go out to the user in a different route not directly on the same site but the email typically goes out on the back end goes to the user's email address where the user is asked to click and then the email is becomes validated now when you user is interacting with your site and if that foreground process the foreground process that you have right there that is actually interfacing with the user is where the user is typing login and password if that foreground process itself becomes responsible for sending this validation email then it has a tendency to slow down the user experience for this user you don't want the user experience to slow and therefore you need to have some kind of a worker process in the background and you can assign these worker processes to send validation emails in the background without having to block the user you don't block them so in the very first example I started talking today I said you know we have a Rails application and then we have a Postgres database and then we might have some kind of a worker process that does work like in the example I gave was sidekick sidekick was a separate container that actually does the work that Rails doesn't want to do it's like a background process for you and the queue itself is stored in a separate container that's the initial discussion we had today and so when the user is interfacing with the application you don't want the user to get bogged down by giving the job of this sending email back to the user for validation to this guy but instead you say you know what worker you please send that email and so the worker sends that email and then Rails will continue to interface with the customer and give them the fast user experience you want them to have but the work is offloaded the work of sending email to the user is offloaded to a separate service or a separate background process think of this as the officer sitting in front and this is the backend worker who's actually printing your social security card or the driver license in plastic and then mailing it to you that's a background process it might take eight weeks eight weeks for you to receive the social security card driver license or whatever you have but the officer is not getting like off the chair and making sure that they print that plastic material right then and there and give it to you because if they do that it might take them two hours and by that time you are tired of waiting at the Department of Licensing and you don't want that bad experience so you want this officer to be efficient so they just give you a temporary card and assign a task for the background worker to actually print that piece of plastic and then mail it to you in eight weeks that's the concept that's what this idea essentially is in the queue you have the workers running here and they will work on whatever the queue is and they will finish that work and then when they're done working they will put an entry in this database and maybe do some actual work like sending an email for example so that's where you have this optional chaining where you want to do this next work after this initial work that you wanted to do the worker nodes will then siphon off to another queue that you know what now that I have done this processing I would like you to actually print out the plastic and mail it to the user this is a separate item the plastic is ready it needs to be mailed it's a separate queue for the mail person that's the idea behind separate queues here input queue and output queue so you receive a request to generate a piece of plastic driver license the worker node will do it and then the actual mailing will happen with a mail person taking the piece of plastic putting it in an envelope applying appropriate postage stamp and sending it out to the postal service that's the concept of input queue and output queue as an example and you can apply this concept to any kind of scenario that you might have sometimes these these worker nodes that are creating the driver license will actually create a picture of this driver license that they create and put that in a bucket so that they might store the data of their their output that they created it's a plastic with an image of the person and things of that nature so the the job that comes in might actually carry the image or the photograph of the user that needs a driver license and that data is available in the job store in a bucket they would use that picture from the user create a driver license and give that piece of plastic out to the mail person and scan an image of the driver license itself and put them back in the bucket that's the kind of scenario of job processing in as it applies to driver license creation for example and mixed in mixed the idea of a person with a machine but at the end of the day it's a process that somebody or some machine has to do and that's where this illustration of batch handling of stuff but you can have a variety of things that you can put together the concept is something that is not critical in time so it doesn't have to be immediately responsive for some work like you want to do the idea of video processing you know you saw that discussion in Netflix but these well not this not this but these machines on the back end these guys are the spot instances they are busy transcoding your video files into other formats based on the queue that accumulates for them and they scale out automatically the spot instances the blackout the processing pipeline they convert that big video into smaller videos for a variety of device form factors and then put those video files back in the bucket that's the same concept these are spot instances you can apply the same idea in a batch process which is what this illustration attempts to show worker nodes working in a batch taking a queue to do the things that they want to be done and then they will do those things and then put the output in an additional output that may be done by another entity another team another person or another worker process you put that in a separate queue and whatever you have worked on you put that save it in a bucket that's the concept moving along the next idea for tolerance in this example the idea is to basically reduce downtime so you are able to handle part able to handle fault meaning it's about high availability so in this illustration what they're trying to demonstrate is this idea that you might have the concept of you know availability zone a and availability zone B like that and you are illustrating in this example a different fashion where you have in the in the in the right side of the screen you have the same concept deployed in a different active versus standby mode so two different modes in the left side you have seen already in the first example that we discussed which was web serving our web hosting scenarios and that was the concept of an elastic load balancer and you have multiple availability zones a and B in your web servers auto scaling and application servers auto scaling and in this illustration on the right half of the screen what they are trying to illustrate is that you can have two separate concurrent infrastructures ready and operating the same exact thing basically you replicate whatever you have in in the active locations so in the active location you have an application running and you have the the back end elements all the way to a bucket where you are storing something are illustrated if some things go wrong in here in this active pipe to the user this end user sees that this particular application suite this entire chain gets down for some reason the fact that you are investing your money in actually creating an entire application stack as a standby in another location like for example here and you have machines out there that constructs basically the same scenario but there is no cross dependency or no sharing of anything like here in the example you have the load balancer actually shares the load between zone a and zone B in this example the other things here the a part is standby it is not active until something goes wrong and when it does when things go wrong you will actually switch over to the standby and so it automatically changes the DNS name from for this elastic IP so this arrow will basically go to that location the standby location and the moment it detects a problem here it automatically goes to the standby location and starts continuing to operation continues its operation separately from that route while this is broken you replace and you replace this machine that gets broken by another one and puts in place of the broken box you remove the broken box put this new one that replaces the broken one and then you can decide to mark this new replacement box as your standby while this one gets marked as your active so the concept will be something like this so you have let's say this was active but then it became defective so you said that this doesn't work anymore in that case you immediately convert this to go that route and then you change the name in your mind in your head to say it is not stand by anymore but it is the active one and at the same time while this is down you replace and put this new box in shape and connect your infrastructure appropriately and make it available to the user but again the user is active there in this chain so that is good so keep it running just relabel this one call it Tanray and you are basically ready for the next disaster that might occur it can occur wherever it wants to we don't know where it will occur but wherever it does the idea idea remains is that you are basically ready for the disaster whenever it occurs that you now have constructed this replacement as your next standby and when the disaster strikes you are basically ready for that and you just you know this was a standby for example this was dead and this was active the next disaster strikes it can strike anywhere here or here depending on where the disaster strikes you at least have to that's the concept behind fault tolerance and so this is how certain companies will actually implement their fault tolerant strategies across multiple clouds so you may not necessarily stick to one particular entity that gives you services for example in this example that you're looking at you have the Amazon's availability zone a versus B here a versus B you can use it that way you can also construct a standby scenario doesn't have to be inside Amazon it can be in another location completely separate completely different like Google Cloud for example and by the way this this kind of a deployment costs money it is actually doubling the cost you have the entire suite of applications running in here as well as there so you saw this illustration that I just live demonstration you have the cloud genius services running in one cloud right now Vulture and you saw me construct the entire stack in a different cloud digital ocean and I temporarily modified my hchost file and I was able to show you that it is actually running here and then I decided to kill all of that but that was just an academic exercise to demonstrate to you however the concept is it is not purely academic in nature it actually gives me the ability to walk away from this cloud and decide that you know what I will not only change this I'll actually remove that entry and go to my DNS and actually point it to this location and if that happens my new cloud is actually operational and I'm good now if I am really really it takes me like 10 minutes you saw me do that in 10 minutes if it is really really time-critical and I cannot even waste 10 minutes of downtime what I then should do is actually run two different clouds simultaneously which will basically double my cost practically but that's okay if I really cannot tolerate even a minute of downtime then I need to do setup like this which is basically fault tolerance you're tolerating a fault somewhere here something goes wrong and all you need to do is just go change the DNS that's it and then come back here and build rebuild and put this in standby and this becomes your active whatever is active and what is standby is decided by what DNS says the DNS says this is active then that's active if DNS points to this location then that is active basically what you are really doing is providing higher level of fault tolerance for your company's infrastructure in one cloud versus another cloud versus another cloud you can do how much money you got really that that's what it boils down to you can keep it running in multiple locations but two is good maybe three not more than that doesn't make many any more sense in making more fault tolerance across like this because doesn't really make sense two is good but remember this thing is that anytime you run concurrent two different clouds you are basically doubling the cost so double the cost but a very good solution from a fault tolerant perspective all you really need to do is change your DNS to point to this location as opposed to this location that's all it takes trip and you're done the DNS changes propagate very rapidly in most DNS high-end DNS services you will find that the TTL value DNS TTL value is pretty short so thankfully time to live values are can I lower or raise the time to live value so the idea is if you know shorter the value of TTL is the faster your the cheaper ones will change five minute period the the better ones or higher performing higher quality DNS actually give you a very short one minute TTL value for example this one does this one gives two minute value if you log in here I think you can see the two minute value the login and I think this one gives you a two minute of minimum number so if you just see any of these guys are automatic you can switch over to two minutes at the minimum so that's the minimum number that I see in this particular DNS service in other DNS I've seen one minute which is the maximum amount of downtime you might run into but again that's not really really down what it takes is for the DNS to switch over and so if your other location is actually down like for example this goes down yes users will not be able to access anything until this DNS which is over and the two minute time to live TTL value is what is the maximum exposure that you are exposing a downtime to the user that's that's what it boils down to and if you are if you have the money double spend the whole thing and make it fault tolerant that's the concept described in this illustration which was here is to improve fault tolerance and high availability and they are showing you in Amazon's example they're showing you this a versus B availability zone but remember you can actually implement across two different clouds it doesn't have to be these things like they're showing you so that's the thing that you want to remember that these are commercial diagrams and they will want to make you think that you need to stay with Amazon cloud that is not necessarily true and nothing wrong with staying with them but just saying that it is not required okay that was the high availability discussion any questions on this before we switch over to the other one here this is big data sets so let's see what it shows you and this is a new diagram apparently but the concept of big data as you probably have known or heard or seen or read essentially is that how many of you know and use big data already I suspect some of you do already use it for those who do not let me draw an illustration to demonstrate what exactly big data does and why is it important so let's go draw the illustration the idea behind big data the core concept is that you have a ton of data coming into your system a lot of it and you need to process it right to process your data that comes in it might take some time before you can take on processing more data so the concept of big data comes into play when you get more data than you can process in 24 hours you still get more data and you cannot even finish processing the data from the previous 24 hours and you get even more data that's the challenge that's one of the challenge this is called the the velocity of data it how much data is being thrown at you and the volume of data how much quantity and what speed at which it is coming to you by volume just the sheer magnitude of data how much you're receiving is bigger than the biggest disk you can have so it won't even fit the velocity meaning how much of data is arrived at your location where you cannot even finish processing before new data starts to arrive and that basically means you cannot complete processing so what do you do with the data that's the the other third concept of third B is variety the data that comes at you is of different type it doesn't necessarily fit into a schema so typically you know databases have a schema in case of big data you may not have a schema the data will just come it can take shape of anything and so when you don't know exactly how do you put in a schema what do you do with it so you have a database like Postgres SQL or MariaDB or MySQL database you have these guys and you have a very systematic structure to store contact information a contact information basically has a name or other I should say first name last name address or rather address line one line two city like that that's the information I have and so I have your name here I store it I can store Mukesh and Shahi and then address and then city and zip code and all that structure it's a very structured method of storing and this is the schema of storing data in these databases so I am receiving data that you know is about Mukesh and I received Mukesh's first name his last name and his address and his city and all that but then I also find out that Mukesh is a great poet and you know here comes his points and there are he's written like 4037 points and so they are also coming at you but your structure cannot store it so what do you do with those beautiful work of art 4037 poems written by Mukesh now what do you do with them throw them away throw them away discard them I don't think it's a good idea right so what do you do how do you handle it where do you store it that's the kind of problem you're looking at and that's where you have this other type of databases which are not these these databases are called SQL databases or structured databases if you will structured databases well structure or schema is first schema is the first thing when the data comes in you want to write it you want to save it and you can save provided it fits a schema the schema is on right operation when you're writing something in the database that's when you need to fit in the schema if it fits the schema then you store it this poems that you have 4037 fine works of art written by mr. Mukesh well they don't fit the schema so what do we do throw it away ouch that's what we will do if it is a method which is schema based writing we have to discard it that's a bad idea to discard useful data based on whatever you receive and discarding such data is actually stupid so where do you save it now and that's the idea which is slightly different type of databases come into play which are not schema on right not so when you write to the database when you save something you need to fit into a schema and if you fit then you write so before you write you need to fit the schema then you can write that's the standard old-style databases known as structured databases now this new type that you may have heard of or seen or used do you know what the name is of these types of databases anybody it's called no sequel which basically stands for not only sequel it is not negative not negative have you heard of this phrase before I suspect you do yep so have you used it I guess Postgres is a no sequel right Postgres has no sequel components by the way you're right Postgres now has no sequel components so it does support this is a unique item that supports no sequel as well so that's a MongoDB Mongo is another example so the idea is the Mongo and Postgres and bunch of others these are the no sequel type databases the conceptual fundamental difference between those and these guys are that you don't have to discard that incoming data because the way you store data is you first thing you do is you save as is without even worrying about the schema don't worry and it lets you save that primarily because of this here it stores everything like JSON so you have name okay and you know line one you have this string the value where you live so his address then you have city this is Redmond like that so you keep going but then now you have poems or the poem one you can write this is my first one and keep going like that you can just dump data in JSON structures pretty much like this and then you got more point number four zero three seven or other four zero three seven and here it is and you can stay store all this in a JSON structure just like this even if you have no idea what to do with this quite yet the lines five and six and all these lines in between you might have more like that so tons of data that comes in you just dump it in a JSON file or that in a Mongo database or Postgres or Hadoop or Cassandra or God knows how many of them they all basically handle this type of structure the idea behind this idea of JSON is that it doesn't limit you at the time of writing so you don't have the place to write these details are you oops you don't have a place to write these things there's no room for it in a classic database you can only store name city and whatever you know the structure that's what you can fit in but you want to you want to store it you can that's the conceptual difference between the no sequel type databases and the sequel type databases fundamental differences right there the concept is that if you have you know data thrown at you that you don't worry about schema schema you worry later at the time of reading from the databases schema on read not on writing first thing you do is write save as is and then when you want to read from the database that you apply a schema and you can modify your schema on the fly at the time of reading and you can say that you know what I just want the first name last name and city so this is my schema so give me first name last name and city and now put this as a lens so you know this is your lens so apply that lens and apply it on the database and it will read the output based on the lens that you provide which is a schema as to how you want to read the database and it will extract the relevant information give it to you that's the way to handle big data in in concept and so back in this picture you have data coming in these guys high throughput parallel upload so first thing they're showing you here is to just dump everything in a bucket just put in the bucket right there s3 it's very easy to handle just upload bulk upload large-scale high throughput whatever comes put in the bucket first thing and then you apply your processing engine and that processing engine as in this illustration that they're trying to show you is this segment here which is this huge block of machines that can either take shape of how to infrastructure that runs processing on this data that sits in your s3 bucket and then once that processing finishes that can take more data out from here from the same buckets that comes in so anything the data is coming in the first thing you do is you store in the bucket and you get to do processing and so you can basically use Hadoop style implementation of easy to machines that in a that work in a cluster fashion that makes the processing scale by itself so if you have more data to work on you can add more nodes to the Hadoop cluster and finish the processing before new data arrives so like overnight for example depending on how big your data you have and then once you have the data extracted you can then store that data out to another storage such as they don't show you in this example illustration but Amazon has a service new service not so new actually it is called redshift that's a data warehouse so this is going to cost you about 25 cents an hour at the minimum you can launch a cluster and this is basically a place to store structured data warehouse so whatever you have processed in terms of information that you extract from this gigantic sources of data that you actually make something based on a query that you want to operate on like this lens that I said just extract from first name last name and city information from the incoming data and by the way the whole thing is intact the whole thing is saved as is it is not discarding the forms it saves the whole thing but at the time of reading I am reading only these things if I want to get all the forms written by all the users I can again construct a different query and say you know what go back read again read the source again do another processing and this time extract all the points also and that becomes a separate query which is a separate read operation that extracts this new information that was otherwise not requested early on but this time the second time I'm reading I am actually asking this extra information since that I my method of saving this information is as is as it comes I did not lose the useful parts of data that was otherwise unable to fit in a regular style database so I stored it everything in a bucket then consume it in a in a Hadoop cluster for example and then the processing is done I can store that structured information inside a data warehouse which was which was this thing called redshift which is a service available from Amazon which basically is a data warehouse in the cloud so you have to pay for it of course but then it gives you Postgres capable it is basically built on Postgres as I understand and Postgres like storage that lets you store structured information based on whatever your analysis is done for your further processing by your business people that's the idea behind data warehouses that's what you would do in a large scale huge datasets that you want to handle basically big data processing that's what you would typically deploy in your company if you have huge sources of data coming in that you want to handle any questions on this idea of data processing as such and or big data concepts from a layoff perspective if you don't want to use cloud you don't have to use these buckets and in that case you need to store that data somewhere and you can store that in Hadoop cluster so if you have a local configuration where you have bunch of machines that operate collectively on a large scale you can these guys by the way together will provide you storage in their file system known as HDFS which is a component of Hadoop and this is available to you you don't necessarily have to use the cloud like this bucket scenario you don't have to use it if you have an internal data center you can totally construct your own Hadoop deployment based on this project and this is the Hortonworks site but you should actually go to the Apache site so Hadoop is available here and through this application you you also receive HDFS which is the foundation of storing a bunch of data like S3 but in your data center so if you have thousand machines you can put them together in a cluster that runs Hadoop and then you get storage along with it by itself thanks to HDFS which is a component of Hadoop right there and that's another way of deploying if if some companies don't want to send the data out to buckets because the challenge with sending data out to the cloud is that once you send the data you're kind of sort of practically married to the data center to that cloud because then once you have like petabytes of information sitting in these buckets here it is really hard for you to take that out of this S3 location and move it to some other cloud that is a very expensive operation that is why you know many companies actually hesitate to move very large data sets in the cloud unless they feel comfortable or unless they enter into a long-term contract with Amazon or any of these cloud providers that's the challenging part I have seen is that people don't really feel comfortable and migrate their big data sets over to the cloud because they want to retain control of the data and they want to hold it with themselves so they don't necessarily move to a cloud scenario primarily for business purposes and for a variety of other reasons like security and things of that nature but security is moved because you know it is nothing specific to cloud that is less secure or more secure it's how you manage your data itself that's that's the key aspect but like example of Expedia they are still experimenting whether they should migrate their eight petabyte of data so they have eight petabyte cluster of Hadoop machines storing in their value office which basically tracks user information at these sites so you have seen Expedia.com you probably have used it they have 40 odd such websites like that and here all these companies that Expedia companies where you can see a bunch of Expedia partners these are all companies related to Expedia and bunch of different global sites that you have 40 or 50 different sites people come here they click they buy they do stuff and the the company here is actually tracking all that information back in a Hadoop cluster and that is what is stored in a cluster in their location they're still thinking whether they should go to the cloud or not the last time I spoke they're still debating whether they should actually go they're experimenting with it but they haven't actually migrated as I understand primary challenge is you know once you go with your eight petabytes and go over to a cloud then you are stuck with them practically even even migrating eight petabytes from your campus location over to a cloud location is expensive operation to begin with. So let's any any questions on this idea this discussion Let's switch over. Ad serving this is like Bing ads or your Google ads so you know them. So the idea essentially is from there are two types of users of this method one of the users is this user that visits Google and then it's not voice but just Google and here you search for something like here so the moment you search for something it should show advertisement that I don't see anything I don't know why but hmm I wanted to see some advertisements for a change but oh there we go so we have ad block enabled that's why I don't see advertisements okay let me disable ad block enabled enabled okay and now we should search again let's see what comes up I should see some advertisement so let me open a different browser and see if I can see advertisement there yeah some ads so here we search cloud genius and this guy mail.com and these guys show me advertisements right so the idea behind this block that you see on this website here comes from Yahoo advertisements this segment is an ad placed by somebody this is another segment who is bidding money for the words cloud genius I don't know why Amazon is bidding money for cloud genius as a keyword but apparently they are and so that's what I searched the keyword was and I see these ads and they are showing these ads inside Yahoo's network if you now compare that back to the location here that's what this visitor is seeing is this image and advertisement in form of a image or a video or some text that associates with it that ad itself is served in this example through CloudFront so if you now look at a classic advertisement so for example here you will see where this thing is coming from so we'll go right click and inspect element we will find that this thing may be coming from the Yahoo network and you can actually see that right here it is coming from this search advertising r.search.yahoo.com slash something slash something some string that is also somehow related to msn.com apparently so you saw these this block that where I'm but I'm looking at right now is this piece where I'm looking at this href segment which is basically showing you that this came from the Yahoo network so if you try to copy the whole thing next track and take it to a notepad this is how it looks like and this is basically coming from the Yahoo network as you can see so if you go back here this segment is constructed out of the Yahoo's website which is what you are looking at in this example where the user is being shown the advertisement which is rendered to this location through a content database network which has the corresponding repository file stored in an S3 bucket and the entire characteristic of this who should receive that ad is decided by your ad serving application that is running in some some auto scaling infrastructure like this example that you are looking at which will look at the user profiles as to what people want to search for what people like that from that region from that country from that location where you and people of that that profile I would say that user profile who are interested in likely seeing what they want to be able to see for example you may have seen this concept once you start searching for say Toyota Prius for example you start this one time and then every time you go you will start seeing Toyota Prius advertisements because you just profile yourself and you somehow communicated with Yahoo that you are interested in Toyota Prius so they will now wherever you go on the Yahoo network wherever Yahoo is serving advertisements they have automatically profiled you as a as a cookie that they placed in this browser that knows that this cookie is interested in Toyota so they want to show you all the advertisements that you can think of based on this profile database that just got created and is stored in this database that that's what these ad servers will look at and then start serving you appropriate content from the repository based on whatever is generated from the computation run through this Hadoop cluster this is elastic map reduce which is Hadoop implementation on the Amazon cloud based on spot instances as you can see these are spot instances so you can they can they can you know grow and shrink but not critical in time so if something goes away no big deal you continue processing it's on on on a batch mode it's not not time critical mode but these are basically analysis done based on users behavior and finding similarities of people like that in that zip code using that location what keyword they're searching for and they will show similar similar ads to those people who may be in the same group in the same profile based on their previous click tracks and they may have accumulated and that's the overall concept and again this media serving our advertisement serving varies from company to company you will have been been doing it one way Google doing it another way Yahoo doing its own way and so every company is different in terms of how they are really serving but at the at the foundation of all it a foundation of it all is the concept that they're profiling the user and you're basically sending them ads to the user like this through your ad serving infrastructure so from a from a top-level perspective if you look at this picture there are some black colored elements like these guys the black colored elements and then there are some white elements here these white elements are people who are requesting to place advertisements so you know I would like to get a click through ad so please put my ad so I will give you what to write what to show and that's what these users are using the white items here these boxes the white colored elements are not time-critical because these are the people who want to advertise they are giving you content to serve and then you will serve the content the advertising content to the users whenever they visit the sites so that's the display advertising or search advertising concept similar to very similar to this idea is this concept called Facebook where you see this idea of open advertisement that should pop up like it is an advertisement again I don't have an ad because of ad block okay so block it again enabled on the site disabled now I should refresh and see some ads and the second one should be an act so here is a suggested course that's an advertisement coming from Facebook and so if you see this thing is basically coming straight from Facebook as an advertisement in my feed and I can click on it and learn more and things of that nature and if I do that it will know that I am interested in e-commerce transactions so that's what this will show me and so it is actually asking me for my phone number so that they can call me but no I'm not interested it's not doing that but that's what you look at that's what you just saw in the example there I am provided with this display advertisement which is using these black elements which are basically ad serving infrastructure which are very critical in terms of their responsiveness to the end user whereas the other elements are not that time critical because at that other interface what you're looking at is actually going to say Facebook and placing acts and so you can potentially place an ad something like here so let's say you will create an ad and this interface is going to the other elements which are these sections so click through request click through servers and the analysis that needs to be done to profile appropriate users to show you right so here you can construct a new ad that shows you local awareness for a geographical reason or geographical region or brand awareness on a global location and then whether you want to consider website clicks have you guys seen such things have you advertised before ever probably not no this is what you look at when the other perspective where an advertiser will go and actually place an ad that's what they see so this is what you're looking at from up if I want to search a specific target zip code for example I can then continue local awareness and say you know for our genius this is the area I want to advertise I can increase the mile radius and say in 50 miles radius so here we go or maybe increase more and say custom and say 200 miles radius evidently toward is not supported so 20 is good I guess so 20 miles so we will go and then further tells me that my potential reaching billion point two customers and I can proceed forward target people with the age group of whatever I want like say for example address age group of 35 and higher up to the age of say 60 that's my target age group if I want I want all the men and women and then I'll say automatic placement and then I set a budget limit the budget limit I can set how much money I want and then set a campaign and give it an ad name so it is giving me some random locations I'll give it some location such as you know identifier so some identifier for this campaign and then I continue forward and I can set my daily budgets to be limited to $50 for example and continue and say that you know I would have put an image it is already suggesting an image based on my profile and so it will place an image like I like this for example it's very smart it automatically populating so you can customize your look and feel of the advertisement that you want to send and then place the order if I place the order it will actually start pumping advertisements and it will charge me $50 which is not what I want to do so I'm not doing it but I'm just showing you that this is possible this is how it works from the click-through request process so the people who want to place an ad will go through a separate route and they will basically look at the profile of the user and serve those ads to the end users that's the concept that you're looking at from a advertising advertisement serving standpoint okay let's go next disaster recovery this is slightly different from what we discussed before in fault tolerance the slightly different idea conceptually similar but slightly different the idea is is different in here because what you are doing is actually replicating whatever you have in your campus in your corporate data center let's say you know we are going here on the bottom right corner about bottom left corner is the corporate data center and in going you have a data center already maybe more than one data center but bunches let's see one one of them so here is one data center that you might have you have some storage you have some application servers and you have these people for corporate users are basically looking at this internal data center that you have on the bottom left corner you have some application servers you have a storage gateway and this is probably all too familiar to Mukesh who manages the data center already right so you have it your server and you know storage and application servers this is your campus this is your corporate data center right there and your internal users are dealing with your data center and this is all on premises so it's on the campus all this bottom left corner what you really want in a case of disaster is to have a fallback plan when disaster strikes in the bottom left corner for some reason disaster can mean anything like light goes off or anything happens something wrong something goes wrong that's a disaster and so the disaster basically means that your user corporate user is not able to access this center the data center they're not able to access the services that's a problem and therefore it is a disaster and so the solution to that is what is being what is being described in this illustration is that you actually use the cloud to recover from disaster what does that mean it basically means that if you have a disaster you switch the corporate user and have them instead of looking at the internal data center point them to the cloud here to a secure connection so this is going to be a VPN connection to a VPC and we will do a VPN exercise next time when we meet is to be able to understand what does it take to have this corporate user think or rather they would not even think they would not even know but they will automatically visit their known locations that like they typically do but the entire thing would actually render the user experience for this corporate user as if it is coming from this location the bottom left corner but actually it will come from the cloud the idea is that you establish a VPN tunnel for this user and then switch over the entire traffic that the user will think that they are actually accessing the data center but divert them through a secure pipe to an Amazon VPC in this illustration it doesn't have to be Amazon it can be any virtual private cloud and in that you go through this gateway that you might have for the VPC and you will basically replicate every single thing you have in your data center and you're doing this already without waiting for a disaster to strike because you know if it is if a disaster strikes and then you start thinking about these things about the cloud and a virtual private cloud where you have your database server and all these things it is already too late so to plan for a disaster you already have to construct a separate virtual private cloud location I didn't ideally basically replicating everything that you have in your campus in your corporate data center you have some disk storage some sand storage some application servers machines some database servers you already replicate the database server over and play those backups securely send them to a bucket and run those backups and restore them through to a database server that you have very similar to what you have already on your campus so you might have Oracle database server inside you will run an Oracle database server in your VPC pretty much of the same type same size in everything and actually back up to a bucket and then restore from the bucket on to the database server in a virtual private cloud and you're doing this even when there is no disaster you're replicating the whole thing you have some storage volumes sand network bunch of hard disks EMC storage I don't know what storage you might have you will start connecting them to a storage gateway that will basically back up this whole storage network that you have over a pipe into a bucket and also create snapshots create Amazon machine images for you store your files that you might have in your file servers wherever they are you will also replicate your application servers just like you have in your data center you will create same things like you have here over here same exact applications running live this is in production mode this is the recovery mode when things go wrong you will switch over you have the production mode database servers and things go wrong you will recover then things go wrong you are okay because all you need to do is imagine that this is all gone and all you do is let the corporate user connect directly to a VPC through a secure connection and we'll talk about how that secure connection actually works and we'll show you how you can make use of there are a bunch of beautiful software available open source that we will use and establish a secure connection for the corporate user to think that they are actually connected to the corporate data center like they were before the disaster and after the disaster everything is the same nothing changed to them except the fact that they have a disaster and the whole thing is down on the left bottom side bottom left and you are actually having the corporate users interface with a virtual private cloud in some cloud services that's the upper right corner portion here and that's the concept of disaster recovery when the disaster strikes you don't have to worry about it because you already planned for every single thing replicating like this the application servers here databases here all the storage back through a storage gate resolution to a bucket and already available for you to restore whenever you feel like the restoration is live so anytime you have a change here it basically plays back that change here and keeps it ready waiting for a disaster that's the idea these things are essentially waiting for a disaster to happen and when that disaster happens to your corporate data center that you actually can switch over your corporate users to the cloud in identical virtual private cloud so that they can continue to operate just like they were doing before while you work on fixing the disaster the disaster will then recover you will bring the users back and everything is like back to normal that's the concept of this picture so we'll continue next time when we meet which is on Saturday and 29th at morning 8 o'clock when we will continue do these remainders and also do some hands-on with some interesting technology that has security and things of that nature and so let's do that at that time any questions at this moment please ask away no questions okay so I will stop recording and stop sharing my