Hey, we are recording our discussion, continuing forward from our Netflix case discussion. And what I will now do is to illustrate a live example of something very, very similar that I have implemented as an exercise for you. And so you will see what I'm going to talk about. First of all, let me describe. So when you have this user, this user, that user is somebody who's, you know, external person like you and me outside the company. They don't belong as an employee of the company. So this is basically you and me. So people like us, customers, end users, and they visit the Netflix site. They go basically through a load balancer into a web server that shows them what they have in store. So that is your front end website, front end. And the actual video rendering, when they select a particular movie, let's say they press the play button and say, OK, now show me. So when the click of the button happens at that time, the control gets handed off from the web servers over to a content delivery network and the media files that are stored here start to flow to the customer in the front end as a window inside the front end. You will now see a live example of the same idea. What I will now show you is on this white sheet of paper. What I will do is first of all describe the concept as to what I'm going to show and then I'll show it to you. And you can actually build it yourself if you like. It's actually fairly easy. What I'm going to show you is something like this. You will have a front end website of some sorts. And this is Imagine the Cloud Genius site. Let's assume that that's the site we were talking about. Now here you see whatever you see and then you might see a media element player of some sort. You can play some video here. That video is actually going to be coming, appears to be coming from this site, but it will come from another site, another source. That source that I will have is a separate service that I created as an exercise for illustration and live demonstration is this different box. It is a proxy pretty much to the CDN. Just like we have this example of CDN and the actual media files coming from here going to the user inside the view that they see from the web server's perspective in an iframe. In that iframe you will see this video actually shows up and the viewer is able to actually play it along. The same idea now in the illustration in this example. You will see on Cloud Genius website actually the staging site, not the actual site, the staging site, which is a slightly different URL. I'll give you the URL also. In that URL you will find that I have a new frame that contains this video element and that is able to play a source coming from some other machine, some other. That some other machine I created as an exercise. First of all, let's go read the exercise itself. Here if you go to just, you will find that I'm going to pass a link for you that shows you how I built that extra service. I will pass that link in Slack chat. You have it there. Slack is starting up. I'll also pass the link in our regular Zoom chat. You can see that there. That link is what you're looking at in this section. That is the set of three, just two files. One to prepare the machine and the other one is to instantiate a container service that runs video service that you will see now. I'll just paste in that link for you. I'll also paste the link for the staging site if you're not familiar with that, which I don't think you are. Be aware of me saying it. I'm going to give you a link, learn.corgeny.us. That's the staging site that we have. I'll open that link now. You will see that it looks very, very similar to our regular slide, except there are certain differences. On one half of the screen, I will actually open the staging site. On the other half, we have the code. This code that you're looking at on the right side is the bits and pieces that I have set up in a separate box. First of all, the preparation steps are here. You can see that what I am basically talking about is these steps in here are the ones that prepare the foundation needed to run the service. The service definition is actually right there in that file here. This Docker Compose YAML describes a service. This is just the preparatory work, needs to get Docker installed, this installed, that installed. It just installs the whole thing and then sets it up. Then in that line number 18 that we have here, we bring up the stack. That stack is line number 18, which is a reference to Docker Compose up, which actually brings up this stack provided this thing above. That is the stack file, a Docker Compose YAML file, this file. It comes down to the machine after the preparation steps and then it brings up the stack. When the stack comes up, it starts to stream a video. That video from here, from that setup that I have, is actually streaming right in this window, in that button. You can actually play it and do it yourself. Just go to that site here, the staging site. You will see this new button and just play on it and see what happens. You tell me what happened, what do you see. After you check it out, just tell me what you see. That's a live thing running right now. It's a video service that I have set up using what you see on the right side. It is embedded in this frame. I'll let you watch it first and then I'll talk about it. Let's see how it's actually implemented. Let's see that. I'm going to erase the ink here to make it clear. Clearing up. We'll now see the service description itself, which is in our Docker Compose YAML file. That's the file we'll now read to understand what we have built there. Here is the Docker Compose configuration, YAML configuration for that. Here, if you look at the beginning point, it's actually very, very simple. There are essentially three services running. Services, one. Service number two. Service number three. You already know this. You've done exercises with it. By the way, if you play this button, please tell me what you see there. There should be something popping up. Tell me what you see when you play this on this site, the staging site that I just described to you, that one. In this section here, what you see is the usual reverse proxy that you have used before. Nothing unusual there. This is the front-end reverse proxy, which will connect to internal services inside that machine. The second thing you have here is a certificate issuer service. This image is a companion to this one. It goes to Let's Encrypt and gets SSL certificates and stores them here and makes them available. We have a domain that we have chosen for this service. What I am doing right now is requesting this companion to get me some certificates for this domain. That's what this companion container does. It gets me certificates for this domain name and this email address. When I receive them, these certificates get loaded in that location. They get used by this guy. That's what you see when you go to this site. You will actually see nothing but just NGINX because it doesn't show anything by itself. It just streams a video. It's a re-streamer. I'll tell you what it means. I like to ask you, what do you see when you see here? When you go there, what do you see currently? Tell me, what do you see? By the way, this is a live exercise. That means this window will disappear. It may not be available later on because I might delete it. That may not be persistent. If you're live, like many of you are, like four of you are, please go watch this and see what you see. Just tell me what you see. I think one of you has landed on that site. I don't know who that person is. Two of you landed on that site. I'm getting notifications. I have a tracker built on that staging site that tracks who's visiting. There's a 24-second video. By the way, it is a live video. It keeps going, actually. It keeps on going. What it does is let me now show you where it is coming from. By the way, it is coming from this container, this third one. This container I have labeled, I call it Restreamer. The name I have chosen is this kind of a name that I seem to communicate the correct behavior of what that container service does, this one. Let us discuss that a little bit before we first of all understand what that concept is. Then we will see how it plays out in the context of what we're discussing. I'm going to erase the ink here. Now, I will also play that video here. Play the Play button, and it shows something. This is one of our previous sessions that we have done already. I have muted it by default. You can unmute it, and then you can pause it, stop it. When you pause it, the video doesn't wait for you. It goes on. This is like a live broadcast happening. I have structured it like a live stream. You cannot actually pause it. If you play now, it will just play from whatever time it is at that moment, and it will play from that point onwards. Now, let us understand some of these concepts that are new to some of you, if you have not seen these things before. This is a fresh new exercise that I built the last session. That's why it is just in GitHub right now. There's no reference to it in our live site, and just one reference, this one, on the staging site, because it is still in construction stage. But I think it works, and I'll tell you how it works and why it works, and what's the point of it, all that. The point of this thing is to produce a platform that can show live videos of whatever I want. Right now, I am playing whatever that video is. You can see that it plays, and so it can refresh and reload and all that. It will show you the video at that point in time, whatever is happening. It's like a live feed. Now, I stop it because I don't want to actually play it. But having said, let's go see on the back end what is going on. Now, let's go see a white sheet of paper to understand what's going on. There are three pieces here that are taking effect right now. One is the site itself. I call it learn.cloudgenie.us. There is that frame that I was talking about. There's a Play button. Now, this frame is rendered by another service. I call that service Restreamer. It streams, restreams something. It doesn't have content of its own. It does not. If you give it something, it's like you have to give it a stream. You have to feed a stream to it, feed it. If you feed a stream to the streamer, you have to bring it out to whoever is listening or looking for. That's the idea. There has to be a feed coming from somewhere. That feed is me. I'm actually sending that feed like that to the Restreamer through FFmpeg. I'm using FFmpeg library to stream out a video file that I have currently stored in my recordings in Google Drive or somewhere. FFmpeg is basically generating a stream and is feeding the Restreamer like that. That's where the feed is coming from. That means there has to be a source. That source currently is that FFmpeg stream. You can see it now here. You can see that I am running the FFmpeg command, and I'm passing an input file, which is basically the recording from 1.21, the day this boot camp started. I think I have a typo there. It's 2.0.11 for some reason. At 2.0.19, January, the first video recording from our session, that series that's going on right now, that one. Then I am copying it over. Video codec as is, audio codec as is, audio-video codec copied over, just same codec I'm using, not changing the codec. You can change the codecs on the fly if you like, and create an output which is an FLV format and stream it to the Restreamer server. That's what you're looking at is the Restreamer service that I just built yesterday. That Restreamer is receiving the stream. It is going on for the last three hours and 13 minutes or so. It is streaming at this bitrate and at 1x speed. The total data transmitted in frame per second and the frame number, that's the detail behind how FFmpeg is actually operating. This is on my computer. This machine, the one I'm using, that machine is itself the source. I'm streaming it out to the Restreamer. If I break this, the video on that side will stop. I'll show you by breaking it, by the way. You will see how it stops and why it stops. It should stop because the Restreamer doesn't have anything to receive, so it doesn't restream anything out. The point of using a Restreamer is that you don't want to be streaming directly from your home computer or your work computer because the whole world will come and hit you if there are lots of visitors. They'll all want to receive from your desktop laptop, and that's not a good idea. You want to have a Restreamer like a repeater in between. That repeater is the one that I mentioned is this device, which is running the code that I gave you in Gist on GitHub. It has basically a Docker composed with three services running, gets an SSL certificate and gets a proxy and runs the Restreamer itself. Three services on that container, on three containers on the box that streams out whatever I feed in and just fans it out to multiple destinations. That's what you will now see when I look at the Streamer itself. I will now connect to the Restreamer and show you what it actually is doing under the hood. Now, let's see here. In the Restreamer, this is the processor running in the Restreamer itself. Here, you will examine what it is doing currently most of the time. Apparently, the stream finished. Oh, okay. Stream looks like it finished. It probably stopped. No, it came back. Maybe there are transient things. Here, you see FFmpeg running a whole lot of that, FFmpeg processing. This processing is basically converting that video coming in into various different formats. I'm doing this just to adjust, dynamically adjust. I'm doing this on real-time because it's a live streaming to live streaming restreaming solution. I'm doing this on a real-time basis, but in case of Netflix, they do it ahead of time. They just store the resulting files in all the formats in a bucket like you already heard me say that, discussed the idea here, which is with that idea. Go back to the Netflix example here. They already store your results in that media repository like here, and they don't have to do the restreaming because they can just take the source from here and dump it out to the user in a stream fashion through content delivery network. What I'm doing is I'm actually sitting here, sending my stream to a re-streamer, and the re-streamer is sending it out. That's the difference in the scenario implementation. Because what I am doing, actually, I have users of this idea that I'm building a new feature at the same time creating a new exercise. Both things happen simultaneously. What I will now do is to break the stream. This video should stop, and we'll now see it, how it breaks and why it breaks and what I intend to do with it. Now, let's go see that. That video streaming that is happening here is going on for the last three hours and 17 minutes. I'm going to break that. Here I broke it. I have nothing going on right now. I will now go look at the top on the actual streaming server, and there I have no FFmpeg process now. They are all exited because there is nothing to do here. It's all DockerGen and usual business container, DChem and all that. It's just Docker running basically. No FFmpeg process because there is no need for it. What I will now do is send a new stream there on that target device, on the re-streamer. I will clear this illustration ink, and then I will start streaming a stream that I want to send to the re-streamer. What I will actually like to do is grab my different source, not that video file, but show you something that you may or may not have seen, probably may not have seen this kind of a demo ever. It's very rare to have somebody show you live. Let's see if I can succeed in that. What I'm going to do right now is ask you to refresh this website, the staging site one more time and try to play, and you will find that it doesn't play anything. That's expected. It will not play because the stream is broken. Now, I will actually stream something very, very different. What I intend to do is actually you should now look at my hands. If you can see me, let's see if you can see me. Is my camera open? No. Now, it is open, but it is too bright. In my hand, I have a phone, iPhone, and I'm going to stream my iPhone onto that site, the one you're looking at, which is... Let me illustrate this point just a little different. Here, on my hand, I have the phone. I'm going to use this device as the source and stream it to the restreamer. You will see the restreamer will stream whatever I see in the camera on this phone out to the site. That's what you will now see. I'm going to start streaming now. I'm going to take a video of whatever I see and just stream it up. You will now see what I'm going to see on that site. You need to refresh it again. Let me start streaming first. Here, you're looking at my iPhone, whatever my iPhone sees. That's what I'm going to show you now. I'm going to start streaming on the iPhone. Now, I will ask you to please reload this website, the staging site. There, I would like you to play and see what happens. It should show you something. I see your tablet. Is that a tablet? Sorry, say again? I see a tablet in your hand. The one on my desk is the one where I work. This is the actual tabletop surface that I write on, which is also a big tablet, 24 inches tablet. Here is my lighting equipment that I'm not using right now and more lighting. Basically, you see what I see. That's what's going on. I'm outside the window and there is other people live here and things like that. The point is I'm using my phone as a streaming source. I'm going to stop streaming now on the phone, which means the stream will break. You will also notice that there's a little bit of delay in this live streaming, about 30 seconds or so. I'm going to stop this iPhone-based streaming right now or just put the phone on charging. That's the best way to handle it. Here, I put it on charging and it will look at me. You will see me on the screen pretty soon. In like 10 seconds, this display will go away and you will start to see me on that browser window as soon as I think that I'm coming now. There you have me showing up. This is live streaming happening for real-life use case. This is like you take a phone and you start to stream to your restreamer and it shows up wherever you want it to show. In this context, I've taken my live stream from the restreamer and I'm playing it here. This is my live video. It has about a 10 to 15 seconds, sometimes 30-second delay. That's the idea behind the concept of streaming. We can extend this globally very, very effortlessly by using a content delivery network. Currently, what I'm doing is I have a restreamer box that you saw that I built using the code that I showed you. Let me erase my notations here. I will now make this window full screen there. Now, let's see what's the idea that I want to discuss is like this. In my current example that I just showed you, by the way, this stream is reaching that restreamer and so restreamer is probably busy in FFmpeg or maybe not. Is it not receiving any? The stream must have stopped or something. Is it still streaming? It stopped for me. I think it stopped. I think the iPhone stopped. You got the point. I'm going to ignore it. Forget the iPhone thing. I'm going to shut it down. It's still streaming. That means the stream is not active anymore. Having seen or understood that idea, let's go back here and discuss some of the higher scaling concepts here. How do you scale such a thing? You have a camera in your hand like an iPhone camera and you want to show it to the world. How do you do that? The world has not just 10 people but hundreds of millions of people. They all show up to your show, whatever live shows happen on Facebook Live or YouTube Live or whatever live shows that you may have seen or heard of is what I'm going to describe to you now as the idea, which is this concept actually is the beginning point of all that discussion. If you have a phone and you're using that as a source and you're streaming to a restreamer, you have the code for it now. This restreamer will take your stream and create formats for your target devices. You can have one, two, three, four different formats and you can see the sample that I think I pasted in Slack chat. I'll repeat that sample paste from extract of code so you'll understand and appreciate what I'm talking about is here. I think that's the best way to show you. Here is how that restreamer actually works. If you look at this segment of code, it is creating one, two, three, four, and five, five different levels of resolution. 720p, 480p, 360p, 240p, and 240p even with a lower bitrate, lower bandwidth. This is reduction in bandwidth, and this is basically a variation of the input stream that I am sending from my phone, from another FFMPEG, from any other source, any other streamable source, that source you receive in this RTMP repeater or restreamer, and you create variants of that. These variants that you're looking at here, by the way, this code is available in open source, and I'll give you a link for this code also. I think I gave you just the Docker Compose YAML, but the whole project is open, so you should be able to see it. This piece, basically the one that I'm talking about transcoding happening through FFMPEG, that's the idea here. This is high bandwidth, this is low bandwidth. If you have a person in India, they'll probably receive this. But if you have a person in a city in India, they'll probably receive a slightly better than the lowest level of bandwidth. Therefore, it will dynamically adjust the quality of video to a higher level according to the level of bandwidth that they currently carry. If the bandwidth is higher, it will be feeding them the biggest bandwidth, highest quality profile, meaning a 720p video will be rendered on their screen. That's the scenario here in this example. What this does is that it is restreaming from the restreamer to the site. In that site, you have a frame and then the video shows here. That restreamer is the bottleneck. If you have a bottleneck like this, then if you have hundreds of millions of people hit that bottleneck, it'll be a problem. The box will probably die doing all this for so many people. You want to handle the load away from the box to somebody else. That idea is what I will describe now. You see that the bottleneck that I have is currently okay because it can handle like 10, 15, 20 people, maybe 100 people, no problem. That's the current setup. But if I want to take it really live on a large scale basis and I want to keep streaming live video feed like Facebook Live to hundreds of millions of people, I can do that. I'll tell you how. If you have the answer to how you would do that, please tell me. I think you should know the answer. You should already know. You should already know that answer because we have discussed this in the past. If you know it or if you remember it, please tell me. That, by the way, is a bottleneck. Just remember that. For a million people, it is going to be a bottleneck. The frame that we have in the site itself, the frame, that is highly scalable setup. But the video element is coming from this silly little server which is the only current bottleneck right now. How do I want to avoid that box from crashing or not showing this video segment at all? The site itself will keep on working and it's in a Kubernetes cluster with autoscales and all that good stuff. That will handle 100 million people. I don't care about that. But video will not work. That's the problem. I want to solve it. It will crash the restreamer because that's when it becomes a bottleneck. What do you think we should do to address the scaling problem with the restreamer? What's the solution? And I think you would know that answer. And if you don't remember that answer, then I think I must have not explained it, which is bad. But I think I did. And I think I'm pointing at it right now with my mouse. And so the answer is right here. It is called content delivery network. Again, just like we had it before. You remember that we discussed one exercise where we had a static site, a simple static site in an S3 bucket. And then we said, you know what? You can put a content delivery layer around it and it will make your site globally available worldwide wherever there are points of presence for that CDN. And so Amazon, what was the name of the CloudFront? This is the name of the content delivery network that I think we discussed. And probably we did an example also. The example we used with CloudFront with a CDN had a bucket. And that's when I mentioned that you don't even have to use a bucket. You could put a server in your garage and have that provide or be the origin to the CDN. And then this garage server through the CDN can handle your global load totally all over the world. Comfortably handle that. It is possible for a machine like that, a simple machine in your garage, as long as it is covered by a layer of content delivery network, it can handle global load. The same ideas again, this time replacing the garage box with the restreamer box. So you have this streaming out from the restreamer. It can do 100, 200 streams. That's the bottleneck right there. You want to go to 100 million people or more, you need to put a CDN in place. And so that's the idea under the hood that I will now show you how to do that. How do you get a CDN and put that in place so that it can handle your restreaming solution? So let's go to some CDN and see what they can do for us. So we'll see, like in this example, they're talking about the content delivery network. The example they're using is the Amazon CloudFront. Nothing wrong with that. By the way, there are many such. All of them operate the same way. So you don't have to use this, but we'll just look at two, three examples, content delivery network examples. This is one and there are more. So we'll discuss a few. So let's go to a CDN again. So the CDN is available at console.aws.amazon.com. And it needs a login. So I am logging in and it needs a token. So I give it a token and then I see Amazon console. Then I look for all the services. And so Amazon is getting big and complex for no reason. And so you have to go look for these services. They're hidden in the all services segment. Somehow you don't see them immediately. They're hiding it in the all services button. And then you scroll down to somewhere in here, you will find a content delivery network and they have a name for it. They call it CloudFront. So we'll go there and let's see what we have. And there, what we want to actually do is look at creating a distribution. This distribution that I am talking about is going to be a bit different than the distribution we have in the past. In the past, we said create a web distribution. And so we said web and get started. And then we did HTTP, HTTPS, and we specified a bucket as the origin and then we put a CloudFront contribution, a web layer on top, and then the CDN became available, scalable globally. So we discussed that, we did that exercise. But that is not what we are doing right now. We are going to do this. And let's read what it says. RTMP, that is real-time media streaming protocol. That's what's going on, real-time streaming from my phone to the restreamer. And restreamer is taking my stream and streaming it again. And they're using the RTMP protocol under the hood. RTMP protocol. This is originally created by Adobe Flash Media Server. I am not using the Flash Media Server. I'm using open source. What I'm using is... Let me tell you exactly which server I'm using. I'm using NGINX with an add-on module to do a RTMP. And so that module is this. So if you take that module and compile it into the NGINX source code. This is one piece of the puzzle. The second piece of the puzzle is NGINX itself. So to get the NGINX source code from here, and take the source code, and then you take this module, which is the RTMP module, and you take both these sources and compile and build. That will get you an RTMP server. RTMP server that I am currently using because I added FFmpeg to it. So it becomes a restreamer server. An RTMP server is just give it a file. It will stream out RTMP. That's what it does. What I'm doing is not giving it a file. I'm giving it a stream. And I'm sending that stream with FFmpeg. So you take my stream, you receive the stream. And then, by the way, before you send the stream out, I would like you to also do FFmpeg and create different formats for different types of devices, different bandwidth scenarios, and then stream them out. That's the restreamer. This is just one part of the puzzle. And so it is a complex project. You have to understand these details to be able to successfully do it. I have done it for you. All you need to do is just examine it and do it yourself if you like, if you're interested in media, live media sharing, streaming, things like that. And it also does recorded media, and you can make your own digital video recorder, on-demand services, and all that video sharing. Basically, you can set up your own Netflix, or you can set up your own YouTube, like Ellen Degeneres has created EllenTube. And here they're using this technology that I just built. So open source. This is like $5,000 of no-one nonsense, so don't go that direction. Just use open source. It's much better, much more open. You can read the code, and that's the beauty of open source. To read the code, you can do whatever you want, modify, add FFmpeg, make a restreamer, basically have party. Yes, I call it a party because I enjoy doing these things. And so that restreamer needs to be the origin, just like a bucket becomes an origin here in the web distribution. In our case of RTMP redistribution, what we want to actually do is from my iPhone, it goes to the restreamer, and then the restreamer becomes the origin for this CDN. That's the concept. This concept allows you to create an RTMP distribution. You must store your media files in a bucket. So that's the idea that these guys require for a file sitting in a bucket. That is exactly what Netflix case or the illustration shows you, a file sitting in a bucket. Let's go see that here. Where is that? Here. See that? A file sitting in a bucket. Files here sitting in a bucket, and then content delivery network is taking them and streaming it out to the user in an iframe coming from this side in the browser window. That's what a CDN is going to do, pick a file from a bucket and stream. That's one half of the scenario. In our case, in my case that I'm talking about, I don't want to go to a bucket because I don't even have a bucket. I have a restreamer that streams live. So that idea is not this idea because I don't want to have my files stored in a bucket because I don't even have them. When I'm showing you an iPhone live feed, I don't know what I will show you in the future because the future hasn't happened yet. That's the idea behind live. That future hasn't happened yet. So I have my phone in my hand. I don't know what will happen in the future because nobody knows. So that's what you were expecting to see in a future is when it happens, that's when you see. You cannot see the future ahead of time. But if you have such a thing, that thing is called re-broadcast. And if you don't know what the future is because the future hasn't happened, that thing is called live broadcast. And this is the case where the future hasn't happened yet, live. And so if you want to do that live, that's what you need to use CloudFront live streaming to create a web distribution. So this is like going back here. But you have to create a live streaming web distribution. So that's the concept that you want to talk about. They still ask you to create a web distribution, which is here, but for live streaming. And there's a subtle difference here in terms of, there's a lot more detail in terms of understanding media from what I call a media specialist perspective for us to figure out the level of detail. If you're interested, I am a media person, so I can handle that. I can talk to you about media in the greatest level of depth because if you remember the product that you use today to communicate with other people, Skype for Business, Link, they're basically live media, live audio video collaboration. So I do that. I used to do that for a living. Now I do that for fun. And so that is my specialty or one of my things that I do. And so we can talk more about this if you are interested, but don't want to go too much detail into one area unless you ask for it. So I'll quickly illustrate this idea from this CDN's perspective, go to another CDN and move into the next case discussion. That's the plan. Unless you ask me to go into detail about the type of and the depth regarding media sharing and collaboration in that particular domain, in which case I will go deeper. I'm not going much deeper right now, but I can if you want me to. Having said, let's go see what happens when you say get started and create a distribution. So we get started to create an RTMP distribution here. You can say my origin domain. You can select a bucket that you have. And so this will take you basically from a bucket point of view and get you to create a rebroadcast scenario. That's the example of RTMP distribution settings. Let's see one more case scenario over CDN. Let's go see key CDN. See if we have an account there. I may not have an account, I suspect, but let's actually read. Key CDN, let's see if I have an account or login. It must have expired. Login, see what happens. Yeah, I think it expired. So I have a negative 7 cent charge. So I need to activate that in order for us to use it. I think it will still let us create what is called A. The account is inactive. So account settings and zone, alias. So none of those actually function. I need to provide money before I can proceed. So this is not a good example. We'll go to another CDN. Let's see Google Cloud CDN, RT, HLS streaming. Let's see what pops up. Yeah. Indeed, it is possible to use Google Cloud CDN for HLS media. And here is a reference architecture, solution reference architecture in this link. So let's see what Google says. So what Google is saying is that if you have a live event, which is what I was having in the iPhone, and so iPhone records a video of a live event that happens, like I show you outside my window, what's going on, things like that, you record it in your iPhone, if you like, a local event record. And then I send it to that re-streamer, which is here. And that's the box, which is going to generate an RTMP stream for us to consume. And then we need to send it to a streaming server. Actually, this is the re-streaming server. So I should erase that. I should say that this is not the re-streamer, but this is just the encoding happening at the software inside iPhone. The software is called Larix Broadcaster, Larix. That's the app I'm using. So that app on the iPhone is doing the encoding for me. It creates an RTSP stream that goes to the re-streaming server that I have. So this is a re-streaming device that I showed you how to build on the Docker Compose YAML. And then you can, of course, have a recording happen. And so recordings can be stored here in the cloud storage on the Google Cloud Platform. You can have a recording stored on the iPhone. You can also have the streaming server also record in the recording module. The recording module are also part of NGINX, so you can just enable recording elements. It will store that in a storage location inside Google Cloud. And then we connect this as our origin server to the content delivery network from Google Cloud Platform. And then they're suggesting external CDN. This is just the interconnect that you're suggesting. So Google Cloud is suggesting to use a different CDN. That is strange, very strange. Google's website, GCP Solutions. OK, Google Cloud Solutions Architecture Reference. And they are suggesting. That is strange, but yeah, it is what they're saying. So I have no other way to read it. But what they're describing is that, simply speaking, this can confuse a little bit. So just pay attention to this idea. The idea is that Google Cloud seems to say that you take your phone, you record it and encode it, and then stream it to a streaming server, store it here if you like, but use our interconnect to connect to a third-party service. This service is a CDN service from Fastly. And what that will do is let you connect, let your mobile browser, streaming player, whatever you have. This Fastly CDN will take it from your streaming, streaming server that I built. So if that's what they're really telling us, they're basically saying that, you know, you don't need Google Cloud because I already have my restreaming server set up already externally in Docker containers. And I have my broadcaster software, Larix is the name, the iPhone app. I don't record locally on the iPhone, kind of pointless to record what I see outside my window. And this is my iPhone as the source of the event. And so I have this, I don't care about this, I have this, I have the restreamer server set up already. So what I need to do is point that to Fastly and just skip Google Cloud altogether. That's what Google is telling us, which is what we will do. So we'll go to see that example from this company, Fastly, see what they do. So I will now go to Fastly, Fastly, which is another CDN, by the way. And so this Fastly, what are they saying? Okay, close. And then CDN. But I don't have an account here. But the point is as simple as this. Like we provide an origin to the box that we have, which is the one I mentioned. I'm going to show you that link one more time. The link of the restreamer, which is restreamer. That website, if you just open, this is what you see. All you see is nginx. However, if you actually go inside and go to HLS and provide the appropriate key for that, it opens up a not found because nothing is streaming right now. Having seen this not found on the actual key where I'm streaming, I'm going to start streaming one more time. And so here is your broadcaster software again coming up. And what you will see now is my hand popping up on this website. That's what I have in my hand. And so that streaming will happen pretty soon. And this 404 will disappear because I am actually going to start streaming. Streaming is beginning. And you can see that the window is really small there. See that? It's going to become big. And that's what just happened. You see the small window actually represented. Hold on, let me just shut the camera here. Let me do one thing again. That small window that you saw, that small window that you're looking at, that window actually is an example of a very, very small resolution here, 426 by 240p. That is what first shows up and then it detects bandwidth. Is the bandwidth available higher? So it grows in width and the size becomes that as soon as it detects that I am in this machine, I have enough bandwidth for us to go to the internet. So it just jumps straight to the biggest size available and it starts to show me that. And that is what happens. You saw that it was showing you a 404 and then you see this. And then probably when I refresh now, it will again show you 404 because the stream has stopped. So when I refresh, 404 comes again because the stream has disappeared. Having seen these details, what we need to really do is take from that endpoint, this endpoint and treat that as if it is a bucket. Now you try to stream something and try to download what you actually see. So you will understand and appreciate once I actually try to attempt to download that thing that I'm pointing at, this one. Let me start another stream and try to download and see what pops up. When I open that, I'm going to start another stream and the streaming started and here it is going now. So the streaming has happened. You can see the streaming FFmpeg activity kicked in on the machine itself. Now I'll go to that site, not this, but the actual origin, restreamer origin, this site, and there I try to load it. And it shows you some video, which is fine. Now let us go see something very strange in a command line. So here is the command line. And what you're going to see now is you probably will, if you know HLS, you will already understand what I'm going to talk about and show you, but if you've never seen HLS, you will see something very, very strange right now. Let's go first of all, try to download Wget, that link. So if you carefully read what I'm going to do right now, I'm going to download something right from that link. This is the file that will actually download when I run that. And so let me run that command to download and see what actually comes out. So now I'm going to execute that. So it downloads and it quickly finishes. I mean, it is so small that it is like probably 495 bytes pretty much. So it's a very small file. Let's go open that file now. So we cat and then say key, that key.file name. So here, now what do you see now is the actual download happens when you first visit the streamer. And here you will see some bandwidth negotiation details. So this file is called HTTP live streaming, also known as HLS. This is a protocol designed by Apple. And it is very small download, as you can see. Just this detail is like 495 bytes. But in there, you will have a reference to an index M3U8. Basically, it's a playlist. It's a playlist M3 format, M3U format encoded in UTF-8. That's what M3U8 is. And so this is a playlist format in which you have the lowest resolution, the lowest bandwidth, and increasingly higher resolution, higher bandwidth going on. And these files are stored inside the restreamer at various locations. If you go and inspect that one, you will see any one of these files, by the way, any one of these five files, you will see that it gives you chunks. Let's go inspect a particular example. So let's see if we can pull up this resolution, the bandwidth, the corresponding file is that file reference. So let's go try to get that file. We'll download it now. And that will help you understand what is going on in HTTP live streaming. So the file is... What's the name of the file is? Key underscore 720. By the way, I'm looking at this file name. And so key underscore 720P, and then 2628KBPS slash index dot M3U8. So let's download that file, see what comes up. You bring down a file, say not found. M3U8, typing mistake. Potentially typing mistake. Maybe not. Maybe not. But we can see these details probably directly in the server. Where is the file stored? That's going to be a challenge potentially. But the point I'm not able to look into quickly, so I'll skip that. The key that I want to really mention is that these HTTP downloads that happen on M3U8, that file extension you saw, M3U8, it brings down chunks, chunks of a large file, like a three-hour video file. It creates chunks and it sends you like a 20-second, 30-second chunk at a time. So you can play along forward, backward only in that window, but the window itself is a moving window. So in the next 30 seconds, it will move forward, and then it will move forward again. And that's how a small window of opportunity for you to go rewind and repeat, and the video flows on a real-time basis. It may be a rebroadcast, it can be a live action, whatever, in which case the future hasn't happened yet scenario. But these chunks are what constitutes the M3U8. So the actual download that happens is actually just plain text files and chunks included in the M3U8 playlist, which is what the player is playing in 30-second window span that you saw in the example. And so what we need to do is basically in a content delivery network like this Fastly or KeyCDN or Akamai or Google or a bunch of others, apparently not Google. Google suggests that you go to Fastly, which is fine. So that's their decision. But the point is to take our source files like M3U8 files and other playlist entries encoded in UTF-8 and extract these chunks and then make it globally available through some content delivery network. And that is what and how you will scale it to hundreds of millions of people if you need to. And this thing is expensive. If you're really talking about 100 million people, well, those people who have a need to show a video to 100 million people have a lot of money also. So it's not that big a deal. But just saying that pricing is expensive. It is very cheap when you're doing it on a small-scale basis. But if you're publishing industry on a large-scale organization, and you can estimate the calculations of cost, and that's how much will it cost you. And so three-year saving will be like $2 million or $2.8 million. So that means this is another saving. So that means the actual cost is going to be more. And that's the idea under the hood behind using content delivery networks to make your video production from an iPhone, what shows the back of your window, outside your window. You can show that to the world if you like, if you have enough money. But by the way, this money and the billing happens, by the way, only if there is a viewer. If there is no viewer, setting this up is actually not that expensive. You can totally set this up on a setup. And if you don't have audience, you will not get that bill, which is fine, which is exactly why you should use it. So when you get the audience, you are ready for it and it will scale. And when you have audience, it is like you creating a new product or creating something and you land on the Oprah show. And Oprah calls you out by name and shows everybody your website. And then the very next day or maybe at the same moment, your site will get hundreds of thousands of hits, maybe millions of hits. Who knows? And then it's probably good business for you. So you can afford the cost associated. So you should always set up these things to be able to scale, even though you may not currently have customers that need that. That's how you should design infrastructure. So this is the Google implementation. By the way, they seem to have created a new site. We should go look at that. And this is the site. So this site is, I'm going to paste a link for you, which is in Slack. And that site seems to be a nice site just for architectural discussion. And so here, this is a new site I just ran into, which is Cloud Solutions Architecture Reference for a variety of use case. So media, gaming, and all things, just like Amazon has, just like Microsoft has, apparently Google also has. So they just created it recently. This is the first time I'm looking at it. So let's go examine it a little bit and see what they have inside media. So media, they have the scenario of transcoding, which we discussed already. They have the scenario of, let's go back a step, live streaming, which we played along live right now. Go back a step. Rendering. This is like showing your video to a target user. Same idea. Basically, all these scenarios of media consumption are similar in nature. Hybrid rendering. The concepts are similar to what we saw. Now we can discuss any of the other types of things here, depending on your interest. And so you should tell me. I asked you two sessions ago to tell me what was the idea of your interest. And so there are lots and lots of examples. Like here is one such example that we did. Give me a second. I'm just closing out on a couple of windows here. This is just one example of media as an example case that we used in our Netflix case discussion. So it is an Amazon sheet and it is a Google sheet. You need to select the next topic. If you don't select, I will select one. So you should think like that direction. You should tell me what the idea of your interest is and we'll go that direction. I think the one comment I had before was somebody asking for this content and media serving also. It is very similar to the other discussion. So we can just mention it in a few minutes before I let you give you time to think about what is the area of your interest that you want to discuss. And so if you don't select something, I will take my own decision. Having said this idea about which is a related idea to the one we discussed, is the idea behind that was media sharing. This is media serving. So similar concept. Content and media sharing, so both ideas shared. The scenario here is that you have a live stream source like my iPhone looking out the window or looking at my desktop. That's the source. That's just one source. You can have our Zoom meeting that we are doing right now. And so you can have an FFmpeg source that I already mentioned, FFmpeg source. That source is here, I think, here. It is a streaming source right now. So I'm going to break that right now, for example, just break it. And then the third type of source can be something like our Zoom video conference that is going on right now, Zoom web conference that is happening right now is also a source, a live source, which does not have anything that we don't know what I will do in the future, what words I will say in the future, even I have no idea. So that's the concept. It is like actually live show happening right now. And so that you can also use as a stream source simply by going to that site and say use a custom live streaming server. It's going to prepare a live stream and stream it to the re-streaming server that I built for myself. And so this Zoom conference that we are doing right now will appear onto that same staging site like here. It will appear. Our Zoom conference that is going on right now will appear, I hope, appears in this window when I play. So let's go first check whether it is actually receiving traffic from our Zoom conference. And so here is your re-streamer. And it seems to be receiving traffic. Hold on. No, it stopped. Check traffic. And it is not. No, it's not receiving traffic, which means this will not show anything. Yeah, it's not receiving traffic. So stop the live streaming and live on custom streaming. How do you set? I think I have to configure the destination, which I haven't done. But that's just another source. I'll configure that and show it to you next time. It takes time to configure the destination. You have to go tell Zoom that my re-streaming server is that machine and the port and the permissions and all that, so that Zoom can then inject its live feed, just like you're looking at on your desktop or your laptop. You're looking at my screen right now. That's the live feed coming from me to you. The same scenario, we can inject that into from a source, which is Zoom, into some location, which is there indicating an example of an Adobe Flash Media Server instance. But again, this is a commercial software that costs a lot of money. Don't use it. There is no reason to use it. Just use nginx and rtmpmodule. So compile it yourself and run it. You save a whole bunch of money. There's no need to use it. I mean, it's there. Probably it is free, even though it might be free. Let's go check the pricing on the Flash Media Server. It might be free. I never bothered to check. But even if it is free, you should not use it. Adobe Flash Media Server is not free. Encoder is free. Server is not free, I think. So there are two pieces of puzzle, encoder and server. Server is not free, I think. Price. So Flash Media Server pricing is like, okay, it's $5,000. I suspected that. So that's the pricing. The encoder is free. The server is not. But again, these are commercial software. You should always favor a known open source software that you use. And there are lots of other people who use it. That is a much better solution these days. No doubt about nginx. I have absolutely no doubt. Big companies use nginx left and right. And so why not use it? It is a very stable, good product. And I have been using it for years. Having said, open source. That's the way. So this becomes your restreamer. And then you can stream it directly to the end user. Or you can go through a content delivery network in the context of what we just discussed early on. That's your live streaming source. Here it can be a content source, not a video source. Not audio video source, but a content source. So this screen talks about content also. So you have some content that files of some sort. So you want to make them available through a content delivery network to the user. So you have a content sharing, content serving. You can serve content. This can be a stored object like a PDF file. It can also be a stored object like a video file. And this scenario is the download scenario. But this video streaming scenario, there is nothing to download. How can you download a video of what I see from my window outside as I show you? There is nothing to download. I mean, you will keep on seeing it. When the stream happens in the future, you will receive the stream. But there is nothing downloadable. Here, a video file, an MP4 file, you can download it through this destination, through the PowerPoint destination. So that's the idea behind content sharing. So a recorded video becomes an MP4 file, becomes a content as opposed to a live stream, which is not available yet. It will happen when it does. So that's when you can actually also stream out to a target user. And you can have also content stored in buckets that you can put that in Amazon bucket and directly serve or go through a content delivery network. That's your choice of content serving as described in this implementation example. Having discussed this, what I want to talk about is to ask you, hey, do you have a preference of any of these ideas that we have described in this sheet of paper in the Amazon Architecture Center or the Microsoft Azure Architecture Center or this newly discovered Google architecture example, solution reference. If you have any of these to discuss, I'll let you take more time to think and we'll come back. I'll not consume our time in going in a direction that you don't necessarily want me to go. So what I will now do is change the topic of our discussion to a different topic. So if you go back to one of our exercises we did a while ago and that exercise is here. You probably have done this many, many times by now, at least two or three. Many of you have done it two or three times. And that exercise has 13 little videos, one minute long. And I'll just go there and I'll repeat that exercise, but not manually. So don't worry about it. We're not wasting time. What I want to do is go to that segment of the exercise and do the whole exercise in an automated fashion. So what does that mean? Let me first of all find that exercise. So I'm going to go to the page and let's begin and let's index page. And then you have, what is that thing called? An automated data center in the cloud, that exercise that you have done it before, I think. And so that is what I'm talking about. When you create a VPC in your data center from scratch, you will hear an Amazon console, for example. And there you go to VPC and then you start dealing with all these details about how do you create a subnet? How do you create a routing table gateway? How do you create a subnet that you have done it at least two times? Many people have done it. That's the exercise I want to do now, but I want to do it repeatably. What does that mean? Let's see what it means. It means something like this. I create a bunch of things here, VPC subnet, this and that. So I create a whole setup and create a bunch of machines and set up my applications and this and that. Complex setup, create a full solution. And that is all ready. And my Amazon cloud is all nice and functional. Now what happens is that the boss comes and says, you know what? Do this whole thing again for the other customer because they don't want to be playing on the same setup. They would like to have their own separate. So you need to create a replica of the same thing. And you say, ouch, because you spent like a three week period setting up every single detail about this and that. And, you know, there are a bunch of details. You know, you've done it. So that is the problem. Ouch. So this is customer number one, and here comes customer number two, and here comes your ouch. So what's the solution? I don't want to have that ouch. That hurts. I do this one time, two times, not going to do it again. It's boring. So what do you think is the solution? Do you know what I'm talking about? The solution, you know, in very, very simple terms, it is control C, control V, copy paste. That's a command C, command V for Macintosh people. That's the solution. So it is as simple as, you know, looking at my Amazon example and doing a copy. And go into another setup in another customer's implementation, and there I say paste. And tell my boss, I'm done. The boss is happy. That's the idea. It sounds like so trivial now that I talked about copy and paste. But then how do you do it? That's the question. So you have a setup, and I use the word copy and paste, right? So copy, paste. You can use another word or another pair of words like that. You can call it, you know, extract information from here into a folder or a file. Extract the information from one cloud design implementation, complete information. You extract that out into a file. And then you apply this file to this second example for second customer. This is your customer number one. You extract it and you apply it. So this is another way of saying the same thing, copy and paste idea. I hope you're with me when I'm talking about copy, paste, extract, apply, you know, or any such thing. You know, you can call it a dump. Take a dump from the whole thing here, like a database dump for database people. Take the dump and you restore. Same ideas in a database context. Copy, paste, dump, restore. Or in another words of saying the same thing, you extract and apply. You extract the information and you apply that information to another setup. So that's the thing that I want to discuss. This is one part. This is the second part of the puzzle that will help you create a replica of the same design here in another customer's scenario, C1, C2. So this is what we are going to talk about. This is done in another service in here and in any other cloud. They all have these services of copying, pasting. They have different names for these services. Let's go see what they call it in Amazon. So let's go see it. Amazon's name is this. And so in here, the words copy and paste are not visible. In fact, they don't use these words copy and paste, but that's what they do. And so in other clouds, they have different names for it. But essentially, they all do the same thing. Extract information from one setup, apply to another setup. Come from here, restore there. That's the concept. They call it different names like cloud formation and things like that. But which of these actions is the copy action and which of these actions is the restore or apply or paste action? We don't know. We have to find that out. So we'll see now. A bit of discussion on this is necessary before we jump into it because we have to understand what these things are. So they're talking about a few things here like create a stack. What is a stack? Create a stack set. We have no idea what that thing is. Design a template. And so there's a designing idea. And then on top of all this, there's one more at the bottom actually here. And that is vaguely similar to what we will call copy. So this is what I would say the copy action or take a dump action or the extract information action. This is conceptually similar. Let's go see why it is similar. So if you read, it says create a something, extract and put that in a file, in a template, in a location, some other location. That's what this file means is this template. So you have to extract the information into a template. That's the idea. So that's what we are going to do. Create a file like this from your existing resources. So you have something existing in your cloud. You want to copy that, right? Take a dump, extract in a template. That's this idea right there, these ideas. So we will do that. And by doing that, what we will see, it will create a template for us. We have our existing resources we have created in our Amazon cloud all over the map and a bunch of things, a bunch of services, so many details, you know, this and that. We see in other machines and configuration and God knows how many things, all of that. Existing resources, you want to capture them and dump it in a template. So that's the action. Now, the other action, which is once you have a template ready, you can then take that template and, you know, just apply. Apply the template to a fresh cloud or fresh setup and you will have a recreated the whole thing. So now let us see what that will do for us in our example here. If you read a little bit, you will find that this action. Let's see what it does. It says cloud formation allows you to do something quickly. But what? It takes a template we provide to get you started or create your own template. So fine. That's what I want. I have a template. Will you let me restore it? That's this idea. Restore or the other idea is apply. Another idea is to paste. So take my template, please, my own template that I have from the step below down there. And I will, you know, restore, apply, paste the template that I create in the earlier discussion here down there, which was to create a template. So there you create a template. And that basically means the copy action. You say the paste action, which is my own template. Yes, that's what I want. All of these things are okay, conceptually fine. But how do I do it? How do I look at my Amazon cloud? By the way, every cloud will have their own method of doing the same thing. They call it different names and tell you what the names would be. And you can find out with other clouds also. And you can experiment with that as well. However, the concepts are very, very similar. And so here at the bottom, create a template needs to happen. You first have our setup. So for customer number one, and we need to create a template. And then the template is this file that I will receive this file. It contains lots of details about this setup. That's what I will create like that. And then what I want to do is the step here, which is called create a stack. Not this, but the other one about that. That step, I will take this file and say, create stack. Now for this thing to happen, we need to have somebody. Let's see this. This is my picture of an eye, human eye. This eye needs to look at what do I have in my Amazon cloud. Completely comprehensively take a look exactly what I have in my Amazon cloud. Comprehensive look. And comprehensively look exactly what I have in the entire cloud through the eye. And then create a template by exhaustively looking at every single thing. It needs to be a very careful eye that looks at every single thing in the Amazon account and creates a dump, a template. But where is that eye? Do we have it? I don't think we have it. So we need to have another way to create such a thing that will then look at our Amazon cloud and see what we got and then create the file. That eye is missing. And that's what we need to build first before we have something like that in place. So that eye, by the way, they don't call it an eye like this, but this is just my way of describing the idea. And so what they call it is this. And so this tool is something that you need to have first in your Amazon cloud that will go look at what else do you have in the Amazon cloud and then create a dump. I'm erasing all of this now. So this cloud former is basically another machine that runs in Amazon cloud. One more machine. So you have a bunch of things already and you run the cloud former and it will create one more machine. So in your example, let's say in your Amazon cloud, you have this, this and that. So you have like machine number one, machine number two, this VPC, VPC and that thing and this thing, all that. That's your collection that you have. Now you will start a cloud former. That former will be the eye I am talking about. It will be another one such machines. And this machine will look at what do you have and will not look at itself because that will be a bad idea to look at itself and account it as if the eye was there to begin with. So it excludes itself from the accounting. This eye is not included in the result. What do you have in the Amazon cloud? That result, they will exclude this eye itself, but the eye will examine what do you have and create a template. So before we actually run this eye, I would like to actually have some resources here. I currently don't. So I need to quickly start some resources. And then I will have the eye created that will look at the cloud I have and create a template. So that's the idea. I will now demonstrate live. I know I currently have my cloud empty because nothing is running here. So we'll go to Amazon cloud and create some resource. How do you create one? Well, you just go create a machine. That's the easiest way. So just go say create a machine and here is your Ubuntu machine and your T2 micro and configure details. And I'm not even using this machine. I'm just having a machine just to have something. So here I review and launch and then I launch. Then I select my keypad and launch. So the machine is now ready. Now, at least I have something to show that machine number. This machine is getting started currently pending. But by the way, this is sample. Let's call it sample. So sample machine is now there in my cloud. At least I have some sample to show you. So this eye machine is now I will create that will look at the machine called sample and then produce the result template. Let's go back to the tool. They call it cloud former. And this thing is available at the bottom right there. And the tool is cloud former. It has strange names. But maybe Amazon thinks the names are good. So that's why they have used those names. But let's go see click. And what they're looking at what you're looking at right now is that they are going to use a template. To create the eye I'm talking about this template. Create the eye. And so you can see the template if you click through that. But we can just examine the template here. It's in a bucket, by the way, stored by Amazon. So we can see the template in the designer or not look at the template right now. Maybe look at the template later on to see what the file itself contains. But we are going to create this eye using a template that Amazon has provided to us. So this uses the cloud former designer to modify an existing template. You can design a template. That's not what we want to do. So choose a template. Yes, that's what I want. Choose this template to create. What I want to create is this thing that they call whatever name they have. They call it cloud former tool apparently. So they call it cloud former. Fine. I'm going to create that template. Sorry. Use the template to create this thing. The thing they call it cloud former. That's the name for the eye that looks at this thing. And there I have a sample. The sample is running. So now we'll start using this template to construct a cloud former that will then look at the cloud and then form a template. Actually, it should be called a template constructor, not cloud former. But I don't work for Amazon and they don't care about me. So that's OK. I'm going to erase my red colored ink and look at this template. This one. I'm going to see what that template actually contains. So I will go to the terminal and then get it down. So it came down. The template is that file. Let's go see the file. Open it up. This is how it looks like. It is basically a JSON file. So you have some details and lots of details. By the way, it's a very long file. It has probably how many lines? I don't know. Let's go see. Count the number of lines are 780 lines. Lots of detail. It's a downloaded template ready to create a former. Ready to create a cloud former, which is what we're discussing here. This is your cloud former. Probably a better name for that would be a template constructor. Just saying. Just think it different. I call it an eye because it looks and then it writes the template. And so that's what I call it to simplify understanding. But having seen that template, at least briefly, this template file that I downloaded from this location, which Amazon has created for us. We will now instantiate a cloud former and I using this template. This one. So now we'll go here and say, OK, please use that template. And next, I would like to call it cloud former, also known as the eye. And then I'm going to create a password. The password will be something that will let me log into this new tool that I am creating right now. I am creating a new cloud former, which is going to be basically another box. In this Amazon cloud and in that box, it let me have a username and password. And that is what we are looking at here. Password and username and a VPC. So I use the default VPC and I need to provide a password and I need to provide a username. So I'll do that right now. The username is Nilesh and password will be one, two, three, something like that. Apparently, they don't let me choose a very simple password like one, two, three, because I have experience doing this. So I need to choose a slightly complex password. So I will create a password right now and say, hey, my password is this. So I generated a new password. I'm going to dump that password there. And so this will now let me use my complex sounding password, which I will save somewhere here. So the password is that. That's the password. And the username is Nilesh. And I will now construct a cloud former. And I will keep this information handy just in case. Having said, I will now go next and then I can choose a key value pair. I can assign an IAM role. This is an optional step. And then you can set some watch for watching and monitoring and getting an alarm if something happens. That is also optional. I'm going to skip that. And you can go next. And then let's read. It tells you that it is going to use this template, which Amazon has created. And by the way, this is a bucket called that name and a file in that bucket. The cloud former template. This description is the templates creates a prototype application. Tools allow you to create blah, blah, blah. So basically it creates the tool that will look at your cloud. And I am calling it an eye. And so there is a cost to it. This cost is free for you, not free for me. So we'll see the cost estimation also. That's a good tool to have. Anytime you provide a template, it will create some things in your cloud. And so you need to know how much will it cost upfront. It's an easy way to find out what it costs. The stack name is this and the password is what I have. And the username is this. And it's going to be in the default VPC. It has no other tags in there. Having seen this, I will now begin creating. And then we'll talk about the cost. Maybe we'll go look at the cost first because the screen will disappear. So we'll go look at quickly the cost details here. So we'll examine. We'll open the cost sheet. And it tells me that the monthly cost of this tool will be $16.84. Fine. If I'm you, you can just select the free tier usage. And so that will bring the thing down to $0. This is going to be free for you if you are running it. And so it is not free for me, but that's okay. Having said that, we'll now create a stack. This stack will create a CloudFormer. I know and acknowledge that it will create IAM resources. So I acknowledge that. And then I create. What you will now see is that it starts to create a stack, like right there. It takes creation in progress and like that. And so this thing is happening. And we'll see what it does. So you will see that the output will give us some results here in the Output tab. Events are happening right now. And it created something called a security group. And then it created a web server and a CloudFormation role and this and that. So a bunch of things are happening. One of those things is complete. But lots of other things are pending and pending completion. It will eventually finish in like five minutes. And so until that time, we will discuss something else. And what we are really doing right now, as you know, is we are creating a tool like this that will then look at our Cloud and then dump whatever we have in our Cloud, simply. And then this template becomes useful for us because it captures our state of our Cloud for one customer and will replicate that state for another customer, another scenario. You just take the template and say, hey, apply. It will basically be the same idea that we are doing right now. You take a template, a different template, the Amazon provided template, as we just saw here, that template, this one that I have, sorry, that I have in this file. That template is created by Amazon. We'll see another template that we will create ourselves and we'll save it. But that template is currently being applied on the left side. We are applying it. Let's refresh to see progress. And progress seems to be happening. And so, yeah, once the progress finishes and it shows all complete, you will see an output. Currently, there is no output found. So when we have an output, we will have access to a machine. And that machine will let us access through this password that I have set aside. And that's when we will get to see how do you take a look at your cloud to create a template. Once you have a template, you can always recreate the resources just like we are doing right now. The difference will be right now we have an Amazon provided template. And that is being used to create a cloud former, which is also known as an eye to look at the cloud. We will use the eye to look at our cloud and to create a template. Now we can use this template and then apply it to another cloud number two. And this eye will look at cloud number one. And so that becomes a very easy process, just like this step. Take a template, apply it like this on the left side to get a former created, cloud former tool created. You can use this template that you create as opposed to Amazon created template. This is you created template, your template. You apply that and you get a replica of the cloud. Number two, number three, any number you like, just apply. By the way, this action, all these actions that you're looking at a browser interface, you can also move in command line. So very, very simple. Just take that file, this file here and say AWS S3 CF apply URL, something like that. There is some command line for that. You can apply that using the AWS CLI, which will be something like AWS CLI cloud formation template. Sorry, I typed too much. So like Google again, Google AWS CLI cloud formation apply template. So here's how you do it. And so you create a stack. This action is the back here, the action. And so create stack action involves something like, where is the example? They don't show you examples right away. Yeah, there it is. So example like this. AWS cloud formation, create a stack, stack name. The name here in my case was CloudFormer AKAI. That is the name I chose. That's the name, template body. You have a file reference to a JSON file, some parameters and some value key pair that you want to pass along, some net specifications if you have any. And say enter. When you enter, you get the output like this, stack ID with resource number. And then you can examine the stack. This action is what is happening right now in the browser window. Same thing. We are doing it in terminal. Same you can do it in browser, just like we're doing it right now. Having seen the CLI application, I will now go back to our CloudFormer tool to see what it is doing. So refresh the screen. And it is still doing. Come on. It should be done by now. That sucks. It's just a machine. Come on. Okay. Let's see what it's doing. So what it's doing is created another instance apparently. You can see it. See the machine I had was the sample machine. And now this machine got created, which is running. And it has this IP address. So you should just visit the IP address. I will do that. The report should come successful, otherwise there is no point in visiting this IP address because the report has to succeed first. So this machine is going to be your CloudFormer AKAI, that machine. And the other machine is sample, those two boxes. This one just got created. And it is running and it has this IP address, internal, that's the external IP address. Copy that external IP. And we'll go back to our Amazon CloudFormation to look at the creation is complete. Nice report. Nice report. Very good. The creation is completed. Events have success. And output is available. The output is a key custom VPC website URL and the value is the link where we want to go. So we will go to that site now because that is our eye, the tool, the CloudFormer. So we'll go and inspect. And this is a bug in Amazon. As you can see, they should not prompt a bad certificate. So this is like a show detail scenario. You view the certificate. The certificate is not validated by third party. Boo. Amazon should be booed for this. This is bad. You view the certificate and it shows you that the certificate is invalid. Because it is self-signed. I don't trust this certificate. But I think this is the right machine. They should also cross check. And this is like basically a bug. They forgot to implement SSL. That's the problem. Nothing else. They forgot to use Let's Encrypt. They have their own certificate authority in house in Amazon. Amazon is a cert authority. They should use their certificate for this setup. But probably somebody is not paying attention. So we are fine. I know the connection is not private. So I don't want to go back. Apparently, I will have to go back. Is that the idea? No. I understand the risk. So I understand the risk involved. And so you can visit the site. So I know the risk. I know. I'm fine with it. So I'm going to visit the site anyway. Not in this browser. This browser is... No. I go to Firefox. That lets me handle things much better. Much better. So we go to Firefox. Firefox is best. So we'll go there. And it tells you the same warning. Advanced. And I say, yeah, I know the risk. Go ahead. It says what? Give me the password, please. Give me the password. So what's the password? There. That's the password. The name is Nilesh. So we type the password. And then the password is that string, whatever that string is. So copy that. Dump it here. And then say, okay. And then don't save. This is the tool that we are looking at. This tool has a bad certificate. And the connection certificate is showing you as self-signed. So basically somebody forgot to properly do the certificate. Whatever. And this is the second time I'm seeing a bad certificate from Amazon. From the same exact exercise. It is a bug. I will report this. They'll probably fix it. But now... Let's see what this tool does. Now that the tool is running. What is the tool supposed to do? Let's go look. The tool was supposed to be the eye that will look at the cloud, in this case, customer number one, whatever you have, and create a template for us. So we will now use the tool and ask the tool to go look at my cloud. That's what it's going to do. That's what I'm going to do right now. I'm going to say, can I read all that and select this Amazon region and then say create a template. Just go look at the region and create a template. That's what I want you to do. So the created template should be available to me. Because that's what I want. That I can reproduce this cloud in another location. So please create a template for me. That's the purpose of this tool. So I will ask my tool to run. I'll say, okay, go to this region. Oregon is where I want to. So you will look at that and create a template. So it's going to now actually inspect the entire Amazon cloud in Oregon. I should call it the Oregon cloud. Because it's a separate cloud, not connected directly with other, not a part of the overall cloud setup. Now here you will see, let me go full screen. So you will be able to appreciate the details here. Let's go see what it does. So what it's doing right now, as you can potentially see, is that it's going to walk us through these steps. Each and everything that you have in your cloud, and at the end will give you a summary. And then you say done. And when you say done, you can then possibly download a template. At that time, the template information is going to ask you to write some text here. So write something here, if you like. You can skip. Not necessary. Description is not necessary. But it's a good idea to write a description. So we just write something here. And say this is the state of my Oregon cloud as of right now. That's the state of the union or state of the Oregon cloud, for example. And so here you can just quickly say that, you know, select all the resources. That's a shortcut. And you can do that. But what if you want to carefully select only the pieces that you want as opposed to getting every single thing? So that's why you can have a step-by-step approach towards looking at every single item in your Amazon cloud and then capturing only the relevant detail in your template, because there are certain things that may not be relevant to other customers. So it gives you the choice to carefully select and then say yes or no to every item. And that's the approach I will take right now. I'll say continue. First thing is DNS. Now, I don't have any of my DNS zones in the Amazon cloud, so this entry is empty. But if you had it, it will show up here. There are no resources found, because I have nothing here. Next, all we have is the sample machine, by the way. So by the way, there is a VPC, so we can select that, the VPC. Is there a VPC network? There are lots of subnets and lots of Internet gateway and customer gateway, and so this DHCP option set. There is a gateway here. There are three or four subnets. So all that need to be captured. So we'll capture all that and say check, check, check. So select all, select this, select that, and then continue. And then we go next and select the network access control here and some route table here. So we grab that and go next. And then there are no elastic load balancers, no elastic IP addresses, no network interfaces, and there is a cloud distribution, apparently. So this is not relevant, or maybe it is relevant to your example of a customer, so you can either select or deselect that, if you like, and go forward. And then you have managed services. I have no managed services, so I go continue. Then you have managed services configuration. I have none of them, so I get continue. If you had it, you select it. If you have Amazon EC2 instances, like I have, I have one called sample. I have another one called, what is it called? It's called the CloudFormer itself. You see, there are these machines that I have in the Amazon Cloud EC2. There are two machines right there. One of them is sample. The other one is the CloudFormer, also known as the eye. So this machine number, brother 2, brother Apple, this number B2BA versus the brother 3, Fox Fox, that machine is not visible. The eye cannot see itself. The eye should not see itself. It should be excluding itself. That's what you see here. You see only this machine, the B2BA machine, the sample machine. This machine is the sample one. That's what you want to select. And it did not even present to you this machine at all. This one did not even show up. And you don't want it to, you know, this is doing right. This is exactly how this should be invisible to the eye itself because it is the eye itself. It can see this one, cannot see this one, cannot see itself. It should not be presenting this instance at all. And that's the idea. You see only one instance. Next. Continue. You see that there are lots of buckets I have, but not all these buckets are relevant to our example. So don't exclude that. Or you can keep some. It's immaterial. It basically lets you decide. You can keep the buckets and continue. And then you have some database subnets that I have created, RDSDB subnet groups. So there are four database subnets, one, two, three. And I can select them. And then I continue to the next step. Then I have some SNS topics defined for MOTIC for some exercise I must have done in the past. I think it's a setup that I tried and that I left in the Amazon Cloud, not deleted, that SNS topic is here. It is not relevant, but I can include it. It doesn't matter. So I've included the template. Next, I will see what I have is security group descriptions. So here are certain security group descriptions. Come on. OK, I think I skipped the security back. So I need to go back a step. And here are the security groups. I can select all of them. And I should not be selecting this iCloudFormer security group. And more importantly, this CloudFormer should not even present to me this choice. This choice should not be even made available to me because it is kind of noise. This is not a part of my original cloud. This is something that the CloudFormer tool created. So I don't want to include it. But I should not be even presented with this choice. But now that they have, I should remember to exclude it. That are my security groups that I am selecting. I have another RDS security group I can select. Then I have a topic policy that I can select and bucket policies that I can have here. A bunch of different Amazon bucket policies. And then I go forward with operational details. So I will select the operational aspects of this. I have nothing in there. As you can see, I don't use Amazon Cloud much. Continue. I now have a summary. The summary is like this. I have a VPC. There are some subnets. There is one gateway, DHCP option set, one ACL, one route table, one CloudFormer distribution, one instance, a bunch of buckets, a bunch of DB subnets, EC2 security groups, RDS security groups, SNS topic, and bucket policies. That's the summary that I have. And what I can do is maybe say that I like this summary. I can say, OK, continue, and as soon as I click the Continue button, I get the template. That's how you create a template. You can see the template right here. This whole JSON file is a very long file. But it basically captures the state of my Oregon Cloud as the description says. The state of my Oregon Cloud as of right now. And it captures the whole detail. Every single detail that you have in that file. You can download this file locally in a JSON file, or you can save it in your bucket. So you can choose your bucket that you want and make it go to that bucket. So let me see if I can put it in a bucket. Which bucket do I want to go make it go? What's a good bucket? Let's see. A bucket which is publicly accessible will be a good one. Hmm. OK, let's go dump into the Cloud Genius code bucket. That bucket will collect the template. So I'll just save it in that bucket. So basically save the template. And you're done. That's it. That's all it takes to run the CloudFormer tool, to go inspect what you have in your cloud. And then it creates a bucket. Sorry, it creates an item in your bucket. So the bucket that I chose was... What was the name of the bucket? I think I chose S3. And in S3 I have a bucket called Cloud Genius Code. So that is this bucket. And there is the CloudFormer template right there. And I can download it. So now it's here. I can open it and say... There is the template that just came down from the bucket. So it contains every single detail that I have in my cloud. And I can just pass it to my friends in my company, coworkers, and just give it to them what I have in my cloud. As simple as that. So what we have done up until now, let's recap. So we had some cloud here. Let's call it C1. And we wanted to take a dump in a template. We did that. But how did we do that? We used an AWS-provided template and applied that to a cloud to create a CloudFormer which acts like an eye. It looks at this cloud and creates the template, puts that in a bucket, and you can download it. And I did. Now that I have this template, I can use this template to apply to another cloud. And I can reproduce what I had here just like that. Same setup. Apply to another example just like you apply this template to a cloud to create a CloudFormer. You apply this template that you dump and then you restore. You copy, Command-Ctrl-C, and you paste, Ctrl-V. That's as simple as that. But doing all this is very, very strictly tied. Everything that you did right now in this exercise was very strictly tied to this company. I mean, that was their tool. I mean, we just saw their tool. We used it. Now that we've used that tool, I need to go back in the Amazon Cloud and destroy the tool. Here's the CloudFormation. We have that formation created using this CloudFormation tool. So we want to, just like a create action, there is a destruction action like here. Delete the stack. What this does is basically destroys every single thing it created. So whatever it did, it will undo, delete. It will start a deletion action, and every single thing that was created will be deleted. You will see that the machine called, by the way, it's already deleting the I, but the sample will not be deleted. It should not be deleted. It did not create the sample through the formation tool. So it will not touch sample. It will destroy only the resources it created. It will leave everything else aside. And so right now, I need to go and kill that sample machine myself, which I just did. So now the sample is also dead, and that's how you clean out. So having seen this example, all of this discussion was very closely tied to Amazon. You cannot do the exact same template and restore it on Google Cloud. You cannot do that. That's a limitation. Another thing to observe is that this thing called CloudFormation in Google Cloud, they have a different name for it. They call it Deployment Manager. It's a slightly better name. But again, all these names are different. So here is the same thing happening in the context of Google Cloud is called Deployment Manager. Operates the same way. Different tool operates just on Google Cloud. Let's see one more cloud, Azure CloudFormation. And these guys call it, what do they call it? Yeah, it's called Resource Template. And so yeah, here it is. Again, very, very similar. Resource Template. What is the example for that? Azure Resource Manager. That should be the name, I think. Yeah, so this is the tool that Azure uses. It operates exactly the same way. All these clouds have the same idea. And so what I like is actually none of these ideas. I'll tell you what I like and I'll show you what I like. And I will use what I like right now. And so that is going to be the next exercise. So here's what I like. And I like this because it is independent. It is open. And it is neutral. It lets you operate on any cloud of your choice. You just write what you want. You plan your and you create your infrastructure as if it is code. So you create your entire Google Cloud, Amazon Cloud, Azure Cloud, whatever you like. And you do that with a bunch of files that you have in your folder. And you say Terraform, apply. I write my design. All these things, write, plan, create, whatever I want to create, all that. Put that in a folder and say Terraform, apply. And so it goes and creates the cloud. By the way, this thing does it with any cloud, not just some. It just does a bunch of clouds. Let's see what providers are supported. Major ones include Alibaba Cloud, AWS Cloud, Azure Cloud, Azure Stack, Google Cloud, Oracle Cloud, VMware, blah, blah, blah. Other clouds. Short, smaller names like Digital Ocean Cloud Stack, blah, blah, blah, goes on a long list. Infrastructure, software. It can configure Kubernetes if you like. So a bunch of things. Basically, you can imagine that it will cover that you have, whatever you have. There's possibly a solution here for every type of service. Monitoring, management, databases, miscellaneous, community, and so go read. I will now do an exercise using this idea. And let's go see the exercise. The exercise is on this location. So let's go recap. What are we going to do? We're going to look at this Terraform tool. By the way, this is open source. It's on GitHub right here. And it is a very, very popular product. 16,000 stars, 4,000 forks, 1240 contributors, updated as late as six minutes ago. 23,000 commits, fairly, fairly popular product, Terraform, open source. And that's what we will be using. And let's go quickly review what it does. What it does, as you already have read briefly, and I'll define what it does, is that it will let you create your concept, whatever you have. It will let you write down your concept in a folder, your design, your ideas. You write, you plan, you create, you conceptualize, you create a folder and write that down here. And this is the idea underneath infrastructure written down in a file or a bunch of files in a folder. And you basically, you are able to represent the state of your cloud in this folder. And what our exercise is going to be is that I have already captured the state of my cloud in that repository that I'm going to use right now. I'll give you a link also. So you can also run this exercise. And that repository contains whatever I have in my mind as the concept that I want to plan and create. I have already written down. It is written in that folder, which you will now see that contains the state of my cloud as I intended to be. So with that understood, I will actually bring that folder down locally, clone it, inspect it, read it, understand it, and then apply. And when I apply, it will go to the cloud, like actual cloud, and create this concept. And you will see it happen. All you need to do is understand what it's going to do, which is just read. Now clone. This is the exercise. It's coming in Slack chat. It is also coming in Zoom chat. And I'm going to need the virtual machine for this. So I'm going to start the VM. And I will clone it inside the VM. And that link basically contains two things right now at the top level. There's an ignore file, which basically contains an ignore list of what should not be captured. There are a bunch of things that should not be captured in the Git repository. Especially pay attention to this file. It should not be captured because it contains credentials. Your Amazon Web Services secret key, key pair, that will go in here. So that is what something that you should not share with anybody. And that is something to be hidden from Git repositories. That's why it is in the Git ignore file. And having seen that, that's the only file at the top level. And so I will now look at the other folder. The other folder contains another folder called Terraform. This Terraform is something that we want to really understand. And I will now bring this whole thing down locally in our VM. So we can read it and make use of it and understand to what it is going to do for us. We'll just clone the folder in our VM like that. And open a terminal and say, let's see if I have it already. I don't think I have it. But Git clone. So the thing comes down. The thing is called ntier. And then I will open this in atom. So we'll see it. In that folder called Terraform, you will find there is a file here. There is a file here called Terraform tf.vars. That file contains, change this two times. You need to change that and provide your cloud key pair. Access key and secret key. You need to provide that there. That is something that I need to do before I can proceed forward with using this exercise. I will do that. Meanwhile, while I do that, there are other things to read. And that is something that can be daunting to read and understand. So what I'm going to do is to actually simplify the understanding of this idea in the most efficient manner. And so what I would like to do is, first of all, make sure that we close down certain windows that are not necessary. And on the right side, I have a folder called hold. That hold folder is basically a temporary location. And what Terraform does, as it will run, is that I will be operating inside that folder. Not here, but here. I will go inside that folder, operate there. And that is the folder that is under question. So it will look at this set of files. And it will basically look at these specific files as Terraform runs. There are also these two folders. We'll talk about them also later on. But the hold folder is kind of invisible to Terraform because it's a temporary folder I created just to hide things from Terraform. What I intend to do is I intend to move a bunch of files from here and into the hold folder so that it just hides it from our understanding. So we'll go begin step by step. What I want to do is move these files into the hold area like that. And what this will do is basically hide it from Terraform. And then we will run Terraform so that it becomes simple. And then slowly bring one of these files at a time back outside, one at a time. Right now, I'm hiding the whole thing. I have just this variables file outside. I can even hide it. So here, hidden everything. And they're sitting in the hold folder. And the Terraform tf.vars need to be changed. So we will create a credential and then provide it to these guys. Terraform needs to know. So here is a key pair. Not this. Not this key pair, but this access key key pair. That's what I want. And so I create a new one. And I show and I copy from here and dump it right there. And I copy from here like that. And by the way, you should not be showing your key to anybody, including me. And so that key pair is pasted. And it is saved correctly. That's a beginning step. Having seen this, I'm going to close this file now because it is not necessary to keep it open. So now we are going to go inside the Terraform folder on the terminal. So like this. And then check, do we have Terraform installed? I think we do. And so it is installed. It is slightly out of date, but no big deal. It will still work. You don't have to update it. You can update. Updating is easy. You just download, basically, from that location. You don't have to update, it will work. Older version also works. They keep updating versions rapidly. So that's why it becomes old. Having said, now we will understand what Terraform does. So I said Terraform. The first thing I would like you to do is to initialize. And so it has no configuration files, it tells you. No configuration files. It doesn't know what to do. So it doesn't have any configuration files because I have hidden every single thing. So what I want to do is to give it some files. For example, I want to open up certain elements here. Say, for example, I want to give it our first exercise. So say, begin with creating an AWS VPC. We're going to create a fresh VPC for us. Without us touching the browser window on the other side, it will just do it for us. You will see it happen. That's the first action. I want to actually bring it outside from here and say, you know what? Out of the hold area, into the Terraform folder. Go. So that is the first step I want to do. I will read that now. So set up our AWS provider. It configures itself to work with AWS. It has an access key, secret key, and region choice. And then you define your VPC. The VPC name will be automated. That's the name of the VPC. Now we go here, examine on the left side in our VPC section. Do we have a VPC by that name automated? I don't think so. We just saw it. We had only one default VPC. We also see that there is this variable region defined that is missing and undefined. The secret key and these variables are not actually called out. They are defined in the variables file. That also needs to be brought outside in the Terraform folder. So now we have the variables also defined. And the variable definition for Amazon region is US West 1. Remember that this is a different region, which means I need to go switch from Oregon onto California just to illustrate this exercise. This exercise is written to work in California as opposed to Oregon. So you can change this easily. And you just change the variable to number 2, it becomes Oregon. You just change the region to your choice. And so that's the variable definition file. It defines the thing that we're dealing with. It also defines a bunch of other things like access key, secret key, region, availability zone, cider block, public cider block for public subnet, private cider block for private subnet, machine images, AMI number for West 1, AMI for West 2, AMI for East 1. So a bunch of region choices are available for machine images that are defined. And so what this thing is doing for us is that it is basically creating what we have in mind. What do I have in mind is what I have written down here. That's what Terraform does is that you write down your plan and you create your infrastructure as if it is code. And that's exactly what I'm going to do right now is I have written down here is what I have in mind. I'm getting my secret key, access key, region, availability zone, cider block definitions, machine images, and define them in variables. And then I have to ask Terraform to do one thing. And that one thing is to define a VPC. The VPC currently is not defined. It is only a default VPC we have here. And there are two. There is a cloud genius VPC. There is a default VPC. It's in a different region, by the way. It did not get picked up in the CloudFormer tool because this is Northern California and not Oregon. In Oregon, we had only one VPC, the default VPC. That's what you're looking at here. I will now switch to California and you will see there's a cloud genius VPC there. We can leave it alone. I'm not touching it. I'm going to create a fresh new one called automated. How do I do that? Well, I'll not do it manually. Terraform, can you please initialize? So it says, yeah, I will initialize myself with the Amazon cloud. And it knows to initialize itself with the Amazon cloud because that's what this exercise describes here. It says provider AWS cloud. The version, that's the version number. So access key, secret key, and region, you need to grab from the variable definitions, which are defined in here and there. And that's where it gets picked up from. It is going to create a resource called automated right on the left side here like that. One more will pop up and we will now see the result coming in as we ask Terraform to, after having initialized, we'll say Terraform, can you please tell me what are you going to do when I run it? When I run Terraform, say Terraform apply is the actual action. Take my design and apply it to the left side. So I'm going to say Terraform, can you please look at my folder and apply that to this cloud? So it will go do it. It will create a new VPC here called automated like that. Automated. It will appear. It will happen. We know that because I have done this exercise many times. So I know it will happen. But for the first time, if you don't know what it's going to do, instead of asking it to apply, you say, tell me what's the plan. What is your plan, Terraform? And so it will tell you the plan. It will actually go and inspect in Amazon Cloud what do you currently have and find out the differences that you have in your document written down on the right side. Make a diff. And it says that there is no thing available there called a VPC by the name that you have specified here. The name is automated. There is no such tag name VPC in here. I did not find it. So I'm going to create one. The plan is one to add nothing to change, nothing to destroy. That's the plan. When you ask Terraform, what is your plan? What will you do? It tells you I will add and do nothing else. I will add this here when you apply. Instead of plan, you say apply. You will see that happen. Let's go do it. It's very simple. By the way, you can run it by side by side along with me as I do it so you can just get a feel for it. So Terraform, apply. It's doing it. That's it. That's all it takes. We'll see. By the way, done. So let's go see now on the left side. What did it do? Refresh. And you now have a VPC called automated with this number that matches here. And it used information provided in your variable definition as the cider block here. And a bunch of other variables were computed as a result of these input provided. So the computed variables are run time computations happen. It has picked up the tag name and applied it right there. That's the idea. Now, you can always go back and change your design. You can say that, you know what? I don't like this VPC. So I would like that to be, you know, modified a little bit. Let's say we can modify something. For example, I want to add one more tag here or just tag this to be something different. Like, for example, something different. Change the tag. That's a minute change right now, right? Just a minute change. I just change the tag itself, call it something different. And then I say here, Terraform, what's your plan? You will see what it's going to do now. It will tell you what it does. It will tell you exactly what it's going to do right now. And it's told you that it is going to make a change in yellow color change. It indicates a change. The update in place. I'm not deleting this resource, not deleting it. Just updating in place, not creating a new one. Just update in place. Modify the tag name automated to modify it to be something different. Awesome. Let's do it. So we'll say Terraform, go and do it. So Terraform, apply. So it does it. And it did. And reported success, which means on the left side, if you refresh the browser window, the tag changed. Something different. Simple idea, right? Now let's go make it complex. We can leave it something different. It doesn't matter. So we need to look at some other things that it did. What is going on under the hood? It is capturing the state of your cloud in that file called Terraform DF state. And this contains exact details of what you currently have in your cloud, according to Terraform. So Terraform managed resources include a VPC, that number, this number. And it has this side of lock and network and route table and security group ID. All these got created as a consequence of the tag name is something different. Prior to this state capture called Terraform.tfstate capture, if you look at before we made a change, it also took a backup of the one before, the Terraform TFState backup. It was a backup plan just before the something different action. The earlier action had a different name. So let's go look at the backup and you will find the name automated there. It kept track of what the name was, what the name is, and you can roll back if you like. So you make a change, you can just roll back comfortably. That's the idea behind these kind of backups in the state. It basically maintains the state of what you have on the left side in a file on the right side here. The state is captured in a TFState file. It basically writes down what it's going to do. And so now you can say, hello Terraform, what is the plan? And so it will go look at the cloud and look at what you have and find the difference. So no change and infrastructure is up to date. So nothing to do because already done. You already did it. You want to change, you already changed it. Now you can again change your design anywhere, not just this, but anything you want to change, you can. And Terraform will apply the changes. That's the best way to handle any cloud because it lets you operate on any cloud. There are subtle differences as to how you are to write. The characteristics and the parameters are slightly different. But once you get the hang of it, it's easy to manage any cloud. That's the reason why I like it. It's up to you as to what you choose. You can choose the native cloud company provided the tools. But I think this one is much better and free and free by that I mean open source, not just zero dollar free. It's also beautiful as in you can read the code. You can modify it. You can make it your code. It is open. It can become your code. And that is what I like about open source. It gives you the freedom, not just about saving money. Yes, it saves money. But there is one catch to this open source free business. I think I should mention that. See, I talk about free and free to me means freedom. People talk about free and free to them means this. This is not what I mean. I think I already said that is not what I mean by free because it is actually not free. It is not zero dollars. You need a lot of time to read the code, to understand it, to support it, to get it to work the way you want. But then it gives you liberation, liberty, liberty, liberty, liberty, liberty, liberation, all those things, French, Spanish words, whatever language that is. I have no idea. But that's what I am looking for all the time. It gives me the ability to do what I want, to give me the freedom I want. It comes at a cost. And the cost is your time. The time can be valuable. So if you don't have time, maybe this is not a good idea. Maybe open source is not a good idea. Yeah, I'm just saying open source, you know, this freedom thing or the Libre thing is like, you know, I have a litter of puppies and the puppies are free. Oh, they are beautiful. But this free means that the puppy is zero dollars free, but that doesn't mean that the cost is zero. The cost of owning a puppy in your home includes carpet cleaning. Includes a whole bunch of things. You know that. And puppies are beautiful, no doubt about it. But they are not free. That is something that I want you to understand. I keep talking about free, but I mean this. It is expensive. It is expensive free. The free is about freedom free, not this free. This is the stupid kind of free. I call it because this is what I feel about it. So this is not what I'm talking about. It is actually expensive freedom free, which comes at a burden, at a cost of you having to learn the whole thing, you having to do and understand source code, manage it, do the whole thing. So if you have the time and inclination and ability and skill and everything that gives you pleasure, of course you should go open source. If we don't have time, just buy a software, let somebody handle it. That's fine. That's fine too, nothing wrong with that approach. But I prefer the open approach. Doesn't mean that I don't like puppies. But then there is a cost to it. Puppies are open source. It's like a puppy. You take them home and you have to manage the puppy. Go to the doctor's office, clean your carpet and all that good stuff. Having said, let's go add some more stuff to this Terraform to make it a little bit more comprehensive. So now I have this VPC created called something different. What I would like to do is bring one more item outside the whole area into the Terraform folder so that we can create something meaningful like, for example, in our something different VPC, we have subnets, none, no subnets. We have a route table, probably default route table. We have Internet Gateway, probably none yet. And so we need to create these resources. But I'm not going to use my mouse on the left side. Instead, I will go look at a public subnet and I will create that. So in order to create a public subnet, I need to have this subnet file moved out to the Terraform folder and then minimize the whole area and then look at the public subnet tf file. And this file describes exactly what I have in mind when I want to create a public subnet in here. I want to create a subnet and I will fill it up, but not using the browser. Instead, using this tool, I would like to have the tool created for me and manage the difference, manage the lifecycle of my cloud for me. And so here you can see what it is going to do when you include the public subnet. It is going to create a resource called a gateway resource. And it is going to create a resource called AWS subnet, call it a public subnet. It is going to use the VPC IDE that you have already created here. It is going to use the CIDR block that is specified in the public subnet CIDR block in your variables file here. It is going to use the availability zone defined in your variables file here. It is going to map public IP on launch, set that to be true because that's what you want. Any machine that gets created in this public subnet, you want that machine to get a public IP and you want that to be true. You want to have a dependency structure defined here that this public subnet depends on you having a gateway. So this is how you define a dependency in line number 12. Is that you say, you know what, this is an object that needs to exist before I create a public subnet. So that is how your dependency defined like that. So you first create that resource and then create this resource because it depends on the Internet gateway default. That thing needs to be first in place before this thing gets created. That's the dependency structure that you have defined in your line number 12, depends on. We tag it, tag it called my public subnet. And then for this public subnet, we need to have a routing table called a public subnet. By the way, this exercise is the same exercise that we have done it before. But manually, extensively in like 13 or 14 different short videos I walk you through, that's the same exercise. Except this time I'm not using the left half. I'm going through the right side and just using Terraform as a tool to reproduce the same setup and extend it a little bit. This exercise, extend that setup to another exercise. You will see those things happen. And so what I'm going to do right now is just read ahead and understand and then apply and read ahead and apply and apply. And that way we understand the whole thing and then we build out the whole cloud in our example. So next thing, what do we have here is a routing table for public subnet. This is the public subnet, which depends on that gateway. And this thing will be creating a route table for the VPC that we have created here. And it will have a routing that goes anywhere on the Internet and uses the gateway as the route. It has a tag called MyPublicRT for the route table. And then associate the routing table to the public subnet. So this table needs to be associated with the subnet. That association between a route table and a subnet is another resource, AWS Route Table Association for public. And that will take the subnet ID, which is this subnet, and the route table ID, which is this route table, and associate the two. That's the association. So you have a gateway, a subnet, a route table, and the association between the two. That's what's going to happen in this example. The association, this association is between the two that you have seen before. So you already understand that idea. So what I will now do is close this window and go back here and say, hello Terraform, what's your plan? Can you tell me the plan? So we will look at the cloud and find out what you have in mind. There will be a difference now because we have added four things. Four things that we have in mind are, create a gateway. So gateway, route table, public subnet, and the association between this and that is here. And the gateway is a necessary thing before you can create this. This subnet cannot be created without having a gateway in place. So that's a dependency structure we have put in already. It creates one, two, three, and four resources right there. Nothing to change, nothing to destroy, just create four items, and then you say Terraform apply, and then it creates it. Now we'll see it. So you say Terraform, please apply, it applies, and the application will happen, and it's doing it. And as a consequence of this result on the left side, you will see that we will have a subnet, a route table called my public route table, and a subnet called my public subnet. And the route table has the association that we wanted to see right there. And it associated with the subnet that you just had, you also had a route defined, the route goes from anywhere to anywhere using the gateway. And that is this route, this one. The gateway is also created by the way. And so route table has a route that lets you go anywhere to anywhere using edit route. And you can see the route goes to any destination on the internet using the gateway that you have chosen here, this one. And that's automatically happened through the software, Terraform did that for us. Let's see what else we have. We have a bunch of other items in the whole area. Let's see what we want to get next. What I want to do is, is to understand that we need to have a bunch of things in place, such as a public subnet, we like to have a private subnet also. So this file is something that we need to have outside in our Terraform folder. So this file needs to move from here to there, from the whole area to there. And that file contains a private subnet. It contains a private route table and it contains an association between the two. Now the thing here is that we need to have this private subnet needs to depend on a gateway, which is not this. We cannot use this one. It's like directly exposing your devices outside on the private subnet. You don't want to do that. So what do you do? Public private subnet, this is your private, this is your public. And so you have a gateway, you have a machine here, you can go connect directly to the gateway. So that's okay. But if you have a machine here, it should not go directly to the gateway, not. So what do we need to have? A NAT. NAT is a machine. So you need to go through the NAT, then outside. That is what is necessary for the private subnet. And that idea is here. You need to have a NAT machine ready. It depends on the NAT. What does that do? It'll create a dependency, which means you cannot just randomly run this file. It'll get stuck saying, you know what, I need the NAT machine in place. Where does the NAT get created? On a public subnet. That's where the NAT goes. Private subnet cannot be created without having this NAT in place. So that's a dependency that I have put in there. So you have a route table, you have a subnet, and you can create an association like this. But it depends on the NAT instance, which we don't have yet. So we need to create one. To create a machine, we need to have obviously have a key pair, right? So we need to have the key pair. And we need to have some security groups so that the machine gets ready. You need to have these files also outside. Before we take the public subnet outside, because it depends on the NAT machine. We need to get the NAT machine ready first. So not this file. Not this file right now. Instead, we want to worry about a key pair. Which key pair? Do I have a key pair here? I don't have to worry about these things, by the way. Because I'm not doing anything manually. I don't even have to worry about my key pair. In fact, you will notice that in my Northern California, I don't have a key pair. I don't care about it. Because this guy will create one for me. It will take my public key and upload it there in the right location. That's this key pair example. What you see it does is it takes my file from this location and creates a public key. And the key name that shows up here. And that's what I want to happen first as a beginning step. Before I can create a machine, I need to have a key pair in place. So I'll say, you know what? Go look at that file in the.ssh folder in that location and just create a key pair. That's the action I want to actually do right now. I'll say, hello key pair file. You come outside of here. Go to Terraform. And then we can run Terraform plan. It will tell you that your key pair is missing. I'll add it. And so that's the looking at the key pair right now in the Amazon cloud. It doesn't find it. It says, I will add this key pair for you. I create one. One to add. I'll say, fine, we'll do it. It does it. And what it does, what I will do now is look at our hold area again and look at our security group settings. This is actually easy to understand because we have played with security group ideas before. So I'm going to just walk you through this fairly easily because it's something that you already understand. I'm going to adjust the font size a little bit. So you can see the whole thing in one shot. There are lots of details. You can clone it locally also. So instead of looking at it in a squinted eye, I think it is better to look at this file on a browser window. So here, by the way, that thing must have finished by now. So I just refresh and I see the key pair popping up. The key pair has happened. We'll now look at our GitHub and look at this security group file in a browser window, which is going to be able to give us some flexibility. So here is the file called security groups. And what this file does is the same file you have on the right side, same on the left side. And what you have here are a bunch of security groups defined. One of them is this one. Another one is this one. And there is one more. I think there's one more. So let's go see what the third one is. On the right side, we'll see it, and it is this one. So the third security group is called web. So we have three groups. And let's go quickly read what they're going to do for us. So on the erasing the ink, adjusting the browser a little bit. And so here what you look at right now is the idea about the default group allows any to any communication, as long as it is within the VPC. And that's the default group. And the other group here is NAT for the NAT device. It allows for me to connect to the machine using SSH, port number 22. It allows this extra port for a new software that we will be connecting, setting up or through automation right now is called VPN connection. And that VPN connection happens through open source software called OpenVPN. You can use $10,000 in Cisco or Juniper or those other high end boxes. You can apply that or use this software, which is open source. And it works just like the other guys actually probably better than the other guys because it gives you flexibility. So open source, OpenVPN. That's what this port number uses. So that's the port I want to make sure that you have a company wide access to your VPC using a VPN connection. That's the connection that we will be setting up through this exercise. It will do it automatically. And the NAT device, the egress going outside, egress is going outside, egress is coming inside. Egress scenario here, you want to allow traffic to go outside. You want to allow traffic to go outside for egress. But for internal, it's any to any connection as long as you are within the VPC. That's this egress idea in the default security group. In here, the NAT security group, we are looking at allowing me to connect through SSH, allowing my VPN connection through OpenVPN software across my company. All the employees should be able to access it provided they have a permission. And then egress is go anywhere you want. That's a NAT security group. The third security group is the web security group, which allows people on the internet to come to a load balancer and then access machines I have inside, which may have containers running inside. And so that load balancer is on the public subnet. These machines will be on a private subnet. And that's the scenario of using the web security group. You will see the details here. It allows AT access. It allows 443 access from anywhere on the internet for incoming routes. And the name is called web. So that's the three security groups you want to create. And you will now move this security group file from that folder, hold folder, to the area outside where it becomes activated. So this file security groups needs to go outside to the Terraform folder. And then we can say Terraform plan or just Terraform applies straight away. So it just applies. It will create three security groups. And you will see that it's happening right now. It will tell you the report as soon as it is done. While it is doing it, let's go read the other ones. The one that we are now ready to do is to actually create a NAT box. NAT box will use the security group structure, the key pair, and set up a NAT machine for us. The security group creation has successfully completed. We have three groups added. The three groups are NAT, default, and web. You can now see them here in the EC2 dashboard for security groups in the Northern California cloud. If you can go to more specific and narrow down to your VPC, that you have the name selected to be something different. In that, if you look at your security groups, you will find there is a NAT security group, there is a web security group, sorry, default security group, there's a web security group created. And this one is a default one given to you by default. So one, two, three groups we created. And those are the three here, one, two, and three. These three guys, the three resources added. Now having seen that, what I want to do is bring this NAT server file outside the whole area into the Terraform folder. And now look at the NAT server file itself, which is this file on the right side. Now here the file will create a new resource. It will create an instance in our Amazon EC2 console. It will use a machine image that is defined already in the variables file that you saw before. It will be choosing a T2 micro machine type. It will use the subnet ID called public subnet, use the security groups called default, and another one called NAT. I think there's one file name called NAT. There is a key name that I will be choosing the key name, which I have already created the key pair, that key. I am disabling source destination check. You remember NAT devices, they need to disable the source destination check. If you recall that from the exercise before, the same idea here. What I will now do is using this image that is Ubuntu image. That's a difference in the last exercise between this exercise is that we are actually using plain old Ubuntu machines, just like you have in the variable defined. These AMI numbers are Ubuntu machines, not Amazon NAT machines. They are not NAT. We need to convert that machine that we get into a NAT box. That conversion is a very, very simple two-line exercise. These are the two lines, actually these two, 20 and 21. These two lines running on any Linux box will convert that box into a NAT, as simple as that. But it does. Here, we are taking a simple box, Ubuntu, and running these two lines on that. Inline provisioning, remote execution, using this key pair that we have in our IDRSA. We are going to create a new machine using this image. This is just plain Ubuntu, nothing exotic, just enough OS. In that, we are going to inject this code, inline execution, remote execution. So, sudo iptables nat postrouting masquerade. This masquerading is the fundamental idea behind NAT devices. It pretends and lies to other devices that it is the one that is asking for the traffic to understand the word masquerade from a previous exercise. The same idea, actually implemented. And then, we are forwarding the traffic. This echo one, line number 21, what it does is echoes the number one in this file. As soon as you have the number one in that forwarding file, it will start to forward traffic. And then, your source destination check is disabled already. It will actually allow the traffic to happen across, through and through NAT. That's the idea behind constructing a NAT. So, you convert a plain Linux box into a NAT device by running lines one and two. Now, on the same NAT box, I'm going to install OpenVPN using this Docker image. The name of the image is, I think, here. So, that's the image I will be using. Let's see that image. That image is Kyle Mana OpenVPN. That is from a person called Kyle. And he has generously shared this repository, which uses OpenVPN in a Docker container. And the corresponding Docker Hub image is this one. It was updated 25 days ago. This is Kyle, and here is his repository in GitHub. This is how Kyle looks like. And he's a smart man. And what he has done is created this Docker OpenVPN repository. And we'll be using that. This has quite a bit of popularity also. As you can see, it's 1,000 forks, 4,000 stars. And so, there's also a commercial service running this software somewhere. I forget what the name of that commercial service is, but that's commercial. These guys provide VPN solution for commercials, $5 a month, something like that. So, if you want that service, I think you could set up your own using Docker, using this method. So, having discussed that idea briefly, we'll be using that image to set up a Docker-based OpenVPN solution in the NAT box itself. Now, that is an academic exercise, so it is okay. But in a proper setup, you will have these actions done separately in a separate box, dedicated box, dedicated to running your OpenVPN software. A separate box is a probably good idea. But I'm just making it simple. And we are using the same NAT box that we are creating here using this image, plain vanilla Ubuntu, and converting into a NAT, and also making it run OpenVPN. It is possible to do it for academic reasons, for practical reasons, you should put this in a separate box from a real implementation perspective. Having seen that, I am now ready to execute Terraform apply one more time. And what this will do now is to create. In our whole area, we still have a bunch of files. And so we need to create the private subnet. Also, we can do that right now. I think we can just pull this file out like that. And it will also create a private subnet because the dependency now meets. The NAT device will be created first, and then the private subnet will be created. So you can see now that when I say Terraform plan, it will tell me what it's going to do. It's going to read the cloud, compare with what I have in my setup, and then tell me that it's going to add four things. It's going to add a private subnet, association, routing table, and a NAT device. The NAT device is that device up here, this segment. So that device will be created. I'll say Terraform apply. And I think we're short of time. So we'll continue the exercise next time. But I'll just take a few minutes and just at least finish this off. The rest of the things are slightly more involved. This, by the way, concludes the exercise that we did the first time with the data center in the cloud setup. That part is finished. We still have some extensions of the same exercise ahead with application servers and some other detail to go around with OpenVPN configuration. See how do you apply in a large-scale company to have your employees with specific permission access your VPC. We can see those examples right in the bin folder. There are defined steps. There will be a cloud configuration that we'll also be talking about. These are extensions of the exercise. We'll do that next time. I'm just concluding this step right now to make sure that we finish through at least a portion that we did already in the manual step-by-step exercise. And that's what's happening right now is creating an NAT machine, updating it, and then it will connect, install Docker, and then install the reactions. We'll take a few more minutes for the machine to get ready with Docker and OpenVPN, and then we will have that machine run not only at an NAT device, but also like an OpenVPN container service for us. That is what is happening right now. We should finish in one or two minutes more, and then we will have an error. The error was NAT instances, script, add a process called Linux, data pipe, broken pipe, broken pipe. Broken pipe is a network error. Written in it, broken pipe. That's the error. It's a network error. Basically, it was trying to download the OpenVPN Kyle Bana image, and Docker Hub puked. It could not download that thing, apparently. That's what happened. It happens occasionally. It was trying to download new image for Kyle Mana, OpenVPN latest, and then Docker says error message. Runtime creation failed. Container Linux starting container process called blah, blah, blah, blah, blah, broken pipe, and disaster happened. It's okay. It's totally automated. You can just destroy the whole thing by saying terraform destroy, and the whole thing gets destroyed. We don't even have to worry about it. We just have to say yes at some point, and it will kill. You have to explicitly say yes. It destroys the whole thing it set up, terraform do kill, kill, kill every single thing that you had, and then report back that it killed. Then you say terraform apply, and then do the whole thing again. It takes like a minute to repeat the exercise compared to doing the whole thing manually, mechanically. That's the idea behind using terraform right now. What I will do now is remember that we are at this stage, and we still have to have the application server, and the Cloud config, and the bin, the open VPN related details here to discuss that. We'll discuss that next time. I think you got the idea behind using terraform, which is you can create and destroy effortlessly if you have a design written down in the terraform format, which is what you're looking at right now in this exercise. You should do this exercise your own, by the way, before we get to the next time. You should just say terraform apply. That's how you do the exercise. The whole exercise gets run in five minutes. Say terraform apply, and the whole thing runs. You don't have to go create this whole folder and all that. You don't have to move around stuff. You just say terraform apply. Go to that folder here, the terraform folder, and say terraform apply. That should do the whole exercise without problem, unless you run into the thing I ran into, which I don't think I'll run again. If I run it again, I will not run into that problem. It's a transient problem. Having said, I will now stop recording, and I will see you guys on Monday at 4 o'clock, my time, 7 o'clock East Coast time, and that's the time. So thank you.