Hello, good morning. How are you guys going? Good, how are you? Doing good, how's it going? Pretty good, pretty good. So we have Ram and David there, hi. I guess Saturday and morning in the winter time or rather fall and winter time is kind of getting difficult. I can tell people are not showing up. I guess they want to watch the recordings. That happens occasionally, you know people are just too tired of attending and they just sleep off I guess. So I can again, I don't blame them, I don't blame them. I totally do that occasionally but not for other people that are involved. Okay, so yeah, so how are you doing? How are you guys doing? Doing great. And did you find a chance to look at that thing that we tried the continuous integration piece the other day? Not yesterday, so yesterday after the class, yesterday was the only one day gap, so it was at work yesterday. So yeah, but we can, that's the plan. Yeah, so what I did was actually investigated that further and I remember that you know you have to go to the Docker Hub itself and very specifically connect this thing for example. You're not sharing the screen. Okay, okay. Let me make sure that I'm recording the video. So I think that's being recorded. Yeah, confirm that. I'm going to also share my desktop, which I just did. And now I will go to that location. So what I'm saying is that the linkage between the build engine, which is this location hub.doctor.com is a image repository, not code repository, right? So this is where you store your images, Docker Hub, right? And so we have these two images here. So I'll just look at one of them. Here you will see that there was those sections are missing. Some sections are missing. If you go to the other one, which is this other repository, then I have build settings and build detail and Docker file references. And it is a source repository link. That link you have to make sure that it is connected. And that is the only difference I was able to find. Once this linkage is there in place between the source repository and you have to build settings. And then you have to say that which particular branch, so here you would be master for example. And then you have to set a trigger. And that is how when active builds will happen automatically on pushes. So this is what we need to enable. I think this is a paid service. This turns out to be a paid service, automated building. So it may not be available to you unless you pay for it. However, there may be some trial. So that's one way to look at in automation or build automation. When you want to push something, you have to have this button. That button doesn't show up for the other account here. So you go to build settings. You have to see if you can find it. Let's try to delete this and create it once again. I'm just deleting that image from the repository and see if we can create a new image. So here create and let us see if we can find out the name for it. And visibility is public and we create. It doesn't let me connect a what is called GitHub connection, source connection. So that's the challenge I think we have. So we have to find out where exactly that connection happens. I have forgotten where it happens. So in this case, you can see that in build settings section, you have to actually specify a source repository. How do you specify source repositories? The query I think you need to do to find out how exactly you would pull them up. So once you have that thing pulled up, you should be able to then specify that the source repository is that GitHub location or that build bucket location. Then come back and specify the branch you want to deploy from like master for example. Inside that repository itself, you should have a Dockerfile. That Dockerfile will be looked up and will automatically get built when a push happens to this source. Those are the pieces that I think you need to put together in order for the build automation and deployment to complete end to end. So you should give that a shot when you get time. Make sense? Okay. So I will also, I suspect this is a paid feature, but I haven't had time to actually look into whether it is a paid one or not. So we'll do that later on sometime. Yeah, sure. Let's go back to what we were discussing. Let's see who else has joined. So let's see my list of people. Who else is available now? Oh, we have lots of people there. Oh, everybody showed up. Okay. David, Dennis, Mark, Mukesh, Ram. Hi guys. Everybody's there. That is sweet. Good morning. Good morning. I was thinking that you will all be enjoying a Saturday morning this time in the bed. But no, people showed up. That's awesome. So awesome. So if you remember, like, you know, many of you were doing something else the other day and we covered a bunch of different details on a variety of scenarios, but I'm trying to remind myself as to what we covered. And so we can continue forward from that so that you can at least watch when the video, when you get a chance to watch the video, the video should be available in that location. As you probably noticed at the very bottom, there is a place for recordings. And that's where you will find the videos for the last session that many of you did not attend. But the videos are right there in this section. So you click on there and you should be able to go to that section. And the last thing that we covered is what I'm trying to recollect. And so this fourth video that you have at the bottom, this one, in which we, I think we were discussing a variety of different architecture types. If you just remember the last one was disaster recovery for local applications. That's what I was discussing, which is this scenario. Disaster recovery for local applications. This one, that's what we covered or discussed at a high level. And we discussed, you know, having a data center in your company, such as the corporate data center, and then how do you replicate the whole thing in a cloud of some sorts, and then have the corporate user automatically fail over when something goes wrong in your data center. When the data center fails, your users in the company will automatically fail over to the cloud. That's the example here. That's the disaster recovery for local applications that you may be running in your corporate center. That's the idea here. So you basically replicate the whole thing that you have in your company over in the cloud of some sorts, and have people just go to the other location, which is the cloud location that you might have, and switch away from the data center in case there is something wrong on this side of yours. So the data center side. So that's a failover disaster recovery scenario. Let's continue along in some of the other interesting scenarios here, which are also there in this illustrations. We have this one, which is, think of it as your Windows Live Sky Drive. What's the new name for it? OneDrive, or Google Drive, or Dropbox, Box.com. Those are the scenarios that we are talking about here. And that is the file synchronization service. So think about somebody implementing Dropbox. By the way, Dropbox themselves have gone away from using a cloud these days, I understand. They have built their own cloud. So Dropbox has gone away from AWS. So that's the story right there. So March and April, Dropbox moved away from the Amazon cloud empire. And so 500 million people who use Dropbox are now on Dropbox's own private cloud. That's what Dropbox decided to go away from. So I think at some point, your economics of a business change dramatically, and you end up in a scenario where you will have that, you know, initially cloud is very, very affordable, very, very, very inexpensive, very cheap, because it's all pay as you go. Pay as you go. And so the price over time equation is very nice. If you think about it, because of pay as you go, the pricing you pay is like really small. And as you grow, the pricing can immediately become bigger. And at some point in time, it may be, there is a crossover point where it may be cheaper for you to build your own cloud. And this is a good strategy provided you know how to run your own cloud. Not every company has this as a core competence. So the idea is if you fit this concept into your core competence, like, you know, for example, Dropbox thinks that Dropbox thinks that, you know, running a cloud is their, their baby, they can do it. Then they decide to actually run their own cloud and they migrate away from Amazon's at some point in history, like in March this year. That's when they decided, you know what, bye bye Amazon, we are not using because we are going to be able to create our own cheaper cloud as opposed to these are the AWS prices. And maybe here are the Dropbox prices or cost to Dropbox for running their own cloud. So at some point they found that, you know what, our pricing, our cost structure for us to run our own cloud, which is the news that you saw is basically this implementation of the conceptual architecture of synchronization. So if you look at, you know, any of these services such as, you know, you use them, I think most of these guys, most of you guys use some services of data synchronization, such as, you know, you know, this thing called OneDrive and Dropbox and all those other people, Box.com, Google Drive, Google Drive. The good thing about Google Drive for you who are not Microsoft, not Microsoft people, is that you have, and by the way, everybody has, but Microsoft people may not be interested, is that, you know, this service Google Drive gives you unlimited data, unlimited data through cloud genius. So whatever your cloud genius email that you have, if you use that email with Google Drive, you will get unlimited storage. As long as Google decides to give it to you, that is a free service, perpetually free service that you may want to use. Most of the other companies will have some limitations on the number of gigabytes that you can use. So that's a usage scenario. But you know, Microsoft people may not be interested because they probably already have a ton of storage from here, from their employer. So you may not be interested. Having said, the architectural illustrations that we have here are the usage model where the idea is, like in case of Dropbox, Dropbox, this application that you have there running that looks like this. Where is that? Here. So here, you can see that the concept is that you synchronize Why does this come? I don't want this one. So Google I'm quitting. I'm looking at this particular application called Dropbox. They have this concept where you store your files. There is an account concept. You can also import some files automatically on when you connect to camera, things of that nature. So you can you can do this by basically enabling this button, automatically pictures will go. If you take a screenshot, screenshots will go. And then you can have a network where you can enable synchronization on a local area network, which is which happens a whole lot faster if you have multiple machines within the LAN. Or otherwise, if you have machines away somewhere else, those machines can also get synchronized. So imagine this usage scenario, you probably know this, at least from a usage perspective, you have some I should not write like a file, maybe like a folder. So you have this folder, where you have, you know, file sitting inside. And you want to synchronize this with the cloud, like the Dropbox cloud, for example, and your other machines. So you have to have basically the same kind of a system, another folder, where you will put the same setup file. So you have this file called A, the file called B, you want to automatically have these files synchronized on a machine number one. So this is machine number one. This is machine number two. And this is the same login, by the way, so same user. So you have the same set of files showing up in both locations. That's the scenario you want to build. And you you can enable this through a centralized cloud service. That's the idea cloud service. In this example, the services Dropbox and they will automatically sync these files for you on your other machines and do for example, or any number of machines that you might have. Similarly, there's other user scenarios, which are, you know, if you have a particular file, which is called a shared file, for example, so you mark a particular file as shared. And you have that in a folder. And in that folder, the the file that you have stored here, you want to share it with another user, another user. And so that user will get access to only that resource. And so it shows up in their computer. This another user's computer because it is strictly shared. That's the idea. So if that's the kind of usage you want to enable, and you want to select and share a particular set of files, and keep all other files private to you and your account and have all of it synchronized nicely across all your machines. That's where such services come into play. So from a user's perspective, there are people who are basically storing files. So the idea essentially is that your user who may be generating new content, such as this phone or the camera that you have number two here, this number two, or that particular item will take a picture that device might take a picture, maybe a phone or some other method of uploading a picture that gets stored straight into a bucket of some sorts. And that is in this example, Amazon says three, and here you have a huge collection of files stored. Now, this same user that uploaded this picture may want to access their own picture. And so they may visit a web browser, like number seven shown here. And on that machine, they might visit this load balancer, which is going to show them their own content stored in this location. So this storage that they have in the files repository they might have that will be shown to them through a bunch of application servers that will automatically scale. And this view is what you see when you go to the site called Dropbox.com and you try to log in. That's the interface that you see when you're trying to access your own Dropbox account. And something is wrong. I think I made a typing mistake. This is a bad site. So I am not going there. Dropbox.com. So I will leave from here because I don't want that thing to be running. And so here a sign in and you can sign in with a different service provider like I'm doing right now. And now it is asking me to type my security key, which I will just touch. So where is the key there? I just touched that security key and it left me in. Now you're looking at my Dropbox from inside. And this is the interface that you're looking at in this example where the user, this user, user, the desktop machine goes to this load balancer and is able to see whatever is being shown to him or her from these application servers, which are basically showing you a view into your own repository, whatever you have. So here I have these files and they are just shown to me. And so here is a file as old as 2008. So I have that file sitting here since 2008 and I have not touched it. But that's the file sitting in some bucket somewhere. And it is shown to me through this interface. These application servers are basically looking at the repository to my account and are exposing that to me through this load balancer, auto-scaling across multiple application servers. That's the consumption model. That's how I'm looking at whatever my content is and I'm able to see what I have in my browser on this machine number seven. On the same interface, if I want to upload something, I can actually take something like upload an item and I can choose some files here and then maybe go to the desktop and send something in. For example, I want to save this book that I have on building real-time data platforms. I can just send the book over and the book goes. So this five megabytes of file, this book becomes available in Dropbox. So what you're doing is this person uploading content to the same site, but that basically goes and sits in the same repository on the backend. And at that time it becomes available to the Dropbox user like this pop-up shows. And it is now somewhere here. So you can see the recent files here on the recent files shown up. That file is now available this one right there. And that becomes available to the user, but that actual file sits somewhere in the repository and it is automatically synchronized with everybody else. Now let us take this other example of sharing a file with other people. So you have these followers that you have. These are file followers, basically a group of people that you want to have them follow the file. So for example, let me identify a picture that I might have somewhere. And so maybe a better neutral picture is what I was trying to find out, not these guys. And so yeah, let's use this book. So I can share this item with people and so I can email them out. And as soon as I email them, let's say I email this guy. But email it to this person and I share. As soon as that happens, there has to be an email notification that goes out. And so this file followers are the people who will watch and observe my file that I just shared with them. And so that email sending machines will basically simple email service will be used in this example to send out emails that notifies this person, the file followers that, hey, there is a file for you. So in this repository, this user has shared one file with you, which this user basically shared that file. And so this book got shared. And if you now go to that user's followers email, they will be able to receive this email notification from simple email service, which is basically an SMTP server. For all practical purposes, this SES thing that you have is a simple email service based on the cloud. And you basically have machines sending emails. This is how marketers send emails. And there is another one that I use in this example that I use this service, SendGrid, to send emails that you may have seen. I will send emails through this platform. Very simple and very similar is that SES platform that sends emails to people. So that's what the machines will do is send out emails to the recipient who will notify them with a token. That token gets generated, the security token that will get associated with that particular file sitting in this location that attaches itself with the item that was shared. And that token is transmitted over to the file follower along with a set of permissions, whether they can read that file or they can modify that file at all. And the data about that file itself is stored in this file metadata store. So from a usage perspective, this architecture is a whole lot simpler in terms of implementation. What we are really doing is those main scenarios that you have. One of them is the user himself or herself stores files F1, F2, F3, all in a bucket. And these files are replicated to all other machines, M1, M2, M3 that the user might have. Similarly, this user scenario number two, user number one, let's say user number one, and now wants to share this file with a user number two. This file is called F2, and that is the shared file. So what they will do is on one of those machines, they might have M1, M2, and M3. One of those machines, the user number one decides to share file F2 with another user, U2. And that is when an email notification gets sent out to U2 with a link and a token that gives them access permissions to this particular object. This object sits in this bucket somewhere. And according to the permissions chosen by U1 for U2, the token will carry appropriate permissions to modify the resource, which is this, the shared resource. That's pretty much it in terms of a simplified view in terms of the architecture for a file synchronization service. If you now compare this service with a service that I use with you guys, and we use this thing called, what is it called? It's called Resilio Sync. And most of you know this as a service that you have used, and I've currently paused it because I am running an active session with you guys on a video stream. But however, this is the place where you, all of you actually receive that workstation. And I can see that it is synchronized with all these users. Right now there are 20 of them who have that particular workstation available to you on your machine. So this workstation that you have, that you have access to, that synchronization happens actually through a very different method. It doesn't involve the cloud at all, as you probably know. That service is actually cloudless. It is not like Dropbox, not like Google Drive, not like OneDrive, but a completely different and also unlimited service. What that service does is a slightly different architecture. It has nothing to do with the cloud. It is purely peer-to-peer. What does that mean? This scenario that we are looking at is called Resilio Sync, which is a new name for an old technology. The old technology was called BitTorrent Sync. And it is actually the BitTorrent protocol. And this protocol is open source. It was created by this person called Bram Cohen from Bellevue. And so our Bellevue area, yeah, right here. So he created this protocol and that protocol has gotten a very bad connotation with it because most hackers and pirates use that protocol to steal content from content creators such as the Hollywood videos and studios. And you've seen Pirate Bay. Pirate Bay is basically built on this protocol primarily that people are stealing stuff left and right. And that is unfortunate that this protocol is being used for theft and piracy. However, the protocol itself is not unlawful. As long as you use it for lawful legal purposes, it is totally fine. It's just a protocol. So what this company does, this Bram Cohen's company, which used to be called BitTorrent Sync or rather BitTorrent.com, what they have renamed rebranded themselves as Resilio Sync. But under the hood, it is still the same protocol. And what in that method, in that application, what they do is, first of all, you would remember that there is no cloud. There's nothing. There's no need for one. How do we run this thing? So imagine my computer. I should not draw clouds. So this is my computer. And in there, I designate a folder. And inside, I have these files. I have this file. I'm representing them by dots. And then I create a share from it. And in this collection, I put that virtual workstation, the workstation that we have. That's why I put it right here. And I decide to share it with the world. So it's publicly shared worldwide. And as a consequence, it generates a link. And I can generate that link for you. You have seen that link already in another fashion. So you have seen it probably in... You've seen this link, and you have used it at least once by clicking this blue colored button that calls, let's say, sync the cloud workstations. If you go look at the link itself, the link is actually a public link that I have put in place that looks like this. So let's go open that link. And here it is. So it's a pretty long link. But at the top level, if you look at the link itself, it is actually a link that goes to some IP address somewhere, and then associates itself with a folder on my computer, right there, and passes a token along. So this is the token, right there. And that token basically generates once I create that token, and I have made this token public. And so anybody and everybody can access this from this location. And what this token does is allows them to access my workstation folder on my computer. That's what it does basically, without having to involve a cloud. So when people click on this button, it takes them to the software, which automatically says, you know what, let me get that workstation for you. And that's what's happening right there. Connect to the folder called workstation right now, but I don't need to do this. So I'm going to cancel it. But you understand that that's what's going on under the hood, where somebody who's clicking on this link is able to connect to my folder right here and copy the whole thing, automatically synchronizes peer to peer. And that, from my peer to them, to everybody else worldwide, is how this thing is actually getting transmitted, is what you see in that protocol, Resilio Sync. And that currently is paused so that no synchronization will actually occur because I am running a live video conference. So I pause it on schedule. So there is a schedule set, for example, here, and I can go and see my schedule. And so I have set that schedule to pause it automatically on Saturday mornings and Tuesdays and Thursdays in the evenings. That's where it automatically pauses when I have a session going. And based on that method, it pauses synchronization, but other times it is running all the time. And that's a very simple, elegant way of transmitting content worldwide and very efficiently without having to rely on some kind of an external third-party service. So this synchronization server that you're looking at, it requires a lot more things on the cloud, whereas Resilio Sync doesn't require any of these. So it actually competes very heavily with services like Dropbox and Google Drive and Box.com and all those companies that charge money for storage. So you know the pricing from various companies. They charge money for these services that you saw in this picture. If you use Resilio Sync, which is open, the protocol is open, the application itself is not open. So that's the unfortunate part. Resilio Sync is not free software. It's not open source software. However, there are other set of solutions like that that are actually open and pretty darn good. They're evolving a little bit. So once they evolve, I will switch over. The name is SyncThing. And that's the thing that syncs. And it is evolving very nicely. I have tried it. It works beautifully, but I have not thoroughly tested it. That's why I'm not switching quite yet. This is open source right here on GitHub. And it does pretty much what Resilio Sync does. And peer-to-peer synchronization, open source. And you just run it, and it does whatever it does for you. That's it. I mean, there's nothing else to it. You can also put the same service running in some NAS device, like network and storage device you might have. Not this NAS, but I should say network-free NAS may be a good one. So here is an open source NAS, which uses some microsystems, free BSD, ZFS-based file system. And they have a SyncThing plugin. The SyncThing is built into this software, which is exactly where I use my... This is how I store stuff for Cloud Genius. So this is a 20 terabyte storage system based on ZFS file system that sits somewhere. And it runs all my storage. Don't need all that 20 terabytes, but I just have it. And that's where I put... This is also another open source product that you may have heard of or used. It is pretty powerful. I have been using it for close to nine years now without downtime, unless there was an electrical outage. In that case, yes, it does get downtime. But downtime, what I mean is that you can have multiple hard disks die on you at the same time. There are, by the way, eight hard disks. And I have seen two of them die at the same time. And I was able to continue read and write operations on the pool of resource. The pool is what a ZFS pool is. This is a file system created by Sun Microsystems long, long ago on the SunSys Sun Solaris operating system. And it tolerates multiple concurrent disk failures. That's the beauty that you have in this solution, which is open source. You should take a look at it if you're interested. It's a pretty high end and open source storage solution, which includes SyncThing, by the way. And so that's a nice thing, SyncThing. And that's a brief on synchronization services without having to use the cloud. This uses the cloud, which is what is commercially available these days from many companies, but newer things are happening, cloudless synchronization. Okay, let's change to the next illustration that we have here, which is we already discussed, I think, quite a bit. This was the media sharing scenario where you had this discussion around Netflix. And in that we had the black colored boxes are responsible for your incoming pipe of videos that come into your company. And those white colored segments that you see all these guys, the white boxes here are the distribution segments that people will be able to watch your content. You've seen this picture, I think, in the Netflix discussion that we had the other day. So we'll skip this and we'll move on to looking at how online games are played. So this is, I think some of you actually have worked in online gaming, MSN gaming, maybe not, but you probably understand online games. So this is an illustration of how a user would use a service that is running some kind of an online game. So if you think about online games, give me some examples of games that you may know or have played. Any game that you can play online, anybody. Some examples I can relate to it better. Oh, you don't play games at all? In that case, you need to skip this. But do you play games online at all? There's that old Halo game. Halo. Okay. So this player, Halo, right? This player is using an Xbox. And in that scenario, Xbox becomes your end user client. It is think of it as the browser. An application called the Halo game itself runs locally on your box, but then it has to connect to some service that Microsoft runs. It's called the Xbox service that is going to be running somewhere out in the cloud. So if you have your Xbox, visit that game itself. Like in this URL, it says mygame.com. That would be the halo game.com, the backend service that renders Halo games. If you look at Halo, this game itself, and you will find that... Where is the game? This is the right thing for it, I think. So this game, Halo, originally created for Bungie Software, Ensemble Studios, and then eventually Microsoft. These companies together even build that franchise called Halo. And this thing runs by itself most of the time, but sometimes it needs to... Like the Halo game player on an Xbox needs to have a person play with them. So the scenario would be that when you're playing with another person, so you have another user using another Xbox and playing with you. You've seen this scenario, right? When a multiplayer game happens and people play together and they shoot something or go after something as they play along. That's what Halo is from a user's perspective. If I want to simplify what a Halo user does, it's a first person shooter, first person shooter game. So you're looking at this user, this person number one, who's using this Xbox running the Halo game on the Xbox, is basically collaboratively working with this other person that is using their own Xbox and playing the same game at the same time. Now, this scenario for it to happen, it requires some third party external service. That is where Xbox services come into play. So you have the Xbox service, which is what these devices will interact with. So this I should call it Xbox service. And you pay for this service, I think, on a monthly or yearly basis, but that's a premium service that Microsoft charges money for in addition to the boxes themselves. So they will sell you boxes and also charge you a service fee for the Xbox service in order for you to play a first person shooter multiplayer game, like this Halo game, for example. So let's go a little bit deeper on what's going on under the hood. So if you have a person here and this person has this Xbox, which is basically like a client device that integrates with the Xbox service. And you have another person here that has another Xbox. And this is also a client device that integrates with the central service. And what they're really doing in terms of integration and interaction is that they are trying to enable a scenario where these users are basically thinking that they are going after some common enemy of some sorts. So you have a game, you have an enemy and you have guns and you're going to shoot the enemy or something like that. So the idea would be that you will virtually construct a play or a game. And in that game, you will identify who else is available online right now to play with me. And so the service will be requested for people who may be available that are interested in playing with you. And so you will get a list of people interested, A, B, C, D, F, Y, Z. And this person's name will also be somewhere in the list. And so this person can then decide to select my friend is available online. Let me go select him. And then once they select, there is a negotiation happening under the hood between this client through the central service over to this client and they acknowledge and handshake that yes, we will play together. And that handshake one that happens, that creates a session. So this session establishes. Once that session is established, they will now start selecting whatever the character is, clothing is, guns and ammunition, whatever else they need to select. But they will collaboratively select the person number one and the person number two, collaboratively select whatever they're doing. And all that transaction is basically happening through and through the central service called the Xbox service. And this Xbox service is basically regulating the sessions and the state of these users together and enabling that play scenario, the play scenario where people, these two people are collaboratively going after the enemy, whatever that enemy may be in that particular game. So you have an Xbox client here, an Xbox client here. That's what is not going to be visible in this illustration, but you have in this example a slightly different type of game illustrated, which does not involve a console like we are talking about this Xbox console, which is represented from a conceptual perspective replaced by a web browser. So you can replace this conceptually by a browser and then you can play the game inside the browser. It's not going to run Halo by the way, but just saying that figuratively saying conceptually think of it as a client device. Think of it as a browser that interacts with a central service like the Xbox service or some other service like this illustration here shows my game.com and these players come together and interact with the game itself. This game interaction is again happening between that service that runs on these web servers that are auto scaling. These auto scaling servers are being shared the load. So what we are doing is that these people are interfacing with the central game location, which happens to be this is my game website, which could be Xbox server on the backend and the people, these players may be using a device such as Xbox or a browser or some other simpler games interface with the game location, the central site, which may be configured to a load balancer to scale across multiple web servers and is able to exchange meaningful information on the play themselves. What are the play is happening that information is exchanged in form of JSON files, as you can see right there and the status of the user as to which point in game that you may be in, where you are currently located. Have you shot a bullet? What is your field of view? Which angle are you pointing at? Things of all those things that the status of the user as the person is playing is continuously exchanged with the game servers. You have a person that may be playing this person P1, maybe playing on the Xbox. They're using the Xbox controller orienting themselves, picking a particular gun, picking an ammunition, whatever they have and they pointing at our enemy and trying to shoot. All that session information is exchanged back with the central service in form of status signals or JSON files being transmitted, communicating back the information so that this entity on the other end will know what this player is doing and vice versa so that the other player will also send similar information that this player will also receive because they're playing together in a session and they will exchange that information through the central service so that they will appear to be collaborating with each other and this information needs to be happening. This information exchange between P1 and P2 need to be happening almost instantaneously and the need for low latency arrives right there and that's where you have heard of users complain quite a bit that it is lagging. You may have heard of this word players complain that the game is lagging quite bad and what is going on under the hood is that the latency in between this endpoint connecting to that endpoint all the way to this endpoint and back, the time it takes in milliseconds for the transaction from this point to go through the server to this point and back is quite large. So if it exceeds like 200 milliseconds or more, then the users will start to complain that it is lagging, the game lags and they don't want to play at that time. Maybe they want to come back later and when it is less laggy that that's when they want to play. So this basically boils down to latency of the network. That's where people request that they have a better internet connection for Xbox games to successfully work through and through with another Xbox through a centralized service where the game actually plays smooth and by smooth they mean when they say smooth actually they mean low latency. Latency is the time it takes for a packet to travel from here to there and back. That's what you're looking at in this illustration where these players need to interact among themselves with the help of a central service. So that's the web serving part. But on the other end, you might have the game states information about the game itself, the log as to who played what and how did they play and who got shot and who got killed and things of that nature. That detail is stored in the form of some logs somewhere. So they're illustrating that you can store those in a bucket or the information about the gameplay itself in some kind of a database. So they're showing you an example of a game database stored here in DynamoDB, which is like MongoDB if you will, for example. Mongo is an open source implementation. DynamoDB is Amazon specific implementation, pretty much similar to Mongo. There are some differences, by the way, that you should know. And that's what they're illustrating. So basically think of it as Mongo when you have the backend database. By the way, very similar to Mongo is Microsoft's database called DocumentDB, which is a relatively new product from Microsoft that competes with this DynamoDB. It's called DocumentDB, where you're basically storing JSON. You take full advantage of SQL as well as JavaScript over document and key value data without the hassle of on-premises or VM-based cloud database. It's a ready-to-use database as a service solution. It is DocumentDB. It is very similar to Mongo, but again, not MongoDB. MongoDB is open source. MongoDB. And this thing is open source. So you know that already. But people like Microsoft here and DynamoDB, these guys are basically creating conceptually very similar applications with certain differences that suit their needs better. And that's what they are creating, by the way. And so in this example, the game database will store in some kind of a Mongo database on the backend. And you may want to run some other analysis on the data that gets constructed. Tons and tons of people play the game. So you have game logs sitting there. You have the game database itself. It's in your DynamoDB, MongoDB setup. You may want to run some analysis on the game plays that have happened in the last day or two or whatever. If you want to do analysis, you can then use this scenario where you have Elastic MapReduce, which is like a Hadoop cluster implemented by Amazon. And you can run Hadoop yourself or you can use the service that they give you. And you may have known of this service already, by the way. So Hadoop is this application that I think I discussed briefly last time. And Amazon has constructed their own service that is called EMR, which is what you're looking at in this illustration that you will probably see it already. You've seen it. It's called EMR. It shows up in one of those services somewhere about data, data related services, analytics services, right there, EMR, which is in managed Hadoop framework. So basically this software running in the Amazon cloud is this. And in case of Microsoft, they have this partnership with Hortonworks that runs Hadoop cluster for BB Insights. That's the name that Microsoft uses for a Hadoop cluster running on Azure cloud, which is they call it HD Insights. It is basically Hadoop running this software running on Microsoft's cloud is this thing called HD Insights. And this is what Amazon has. This is what Microsoft has. And everybody has basically implementing this product, which is the underlying technology, which actually comes from Google. So this gentleman, Sanjay Geymavat, he created this thing called Map and Reduce. That's the paper that he wrote long ago in 2004. And that's what has been reverse engineered and implemented in this product. And implemented in this service and this service. But its origins are right there. And you can see that this paper, where is the paper? I think I should give you that paper link to that paper should be here. No. So, Map, Reduce, Geymavat. I think you should read that paper. It should be a nice one. 2004. There. This paper is what he wrote with Jeffrey Dean and Sanjay Geymavat. So I'm sending that over to Slack chat. Come on, Slack chat. There we go. So that's what is the origin behind data analysis software, Hadoop. The origins come from here. And so that's the paper. And this is the reverse engineering application. And this is an implementation in Amazon's cloud. This is an implementation in Microsoft's cloud. And this is what they have in usage in this illustration right there. Map, Reduce. That's what is used for analyzing data of game plays. So this will basically generate information about the game players themselves. And also send communications that relate back to the game players themselves through this service, which is SES, Simple Email Service. This kind of service is basically send out emails, email notification related to gameplay that you may have of some sorts. So people like, there are other interesting games like Microsoft has recently acquired a game company called Minecraft. Is that right? Is that Microsoft game now? I suspect so. I don't know for sure. Some game company, if it is not Microsoft, some other acquisition happened. Is it Microsoft? I don't know for sure. Yeah, it is Microsoft. Microsoft is bringing Minecraft to the Apple TV. So Microsoft buys Minecraft for $2.5 billion, but founders will leave. That's the story in 2014. So this is another good, very famous game, which is the game called Minecraft. You have other simpler games like Amazon. So here, a bunch of free online games that you may have seen. And these are simpler games as compared to Xbox games, but you're playing Solitaire of some sorts, or you're playing some other games, all these listed top 10 games. So you have Mahjong and just words and play with words and bunch of other games that you will have. They can all be implemented conceptually in a similar fashion that you're looking at here, where you have a bunch of web servers running the actual game, helping the people play those games. And these players play it together among themselves with the help of this central service and information about that gameplay itself is stored in centralized locations, such as these illustrations show. And that's pretty much from a from a gameplay perspective is what you have in this illustration. Let's move on to another one. And I want to finish these guys quickly. So you have one, two, three, four, five, okay, five, five or six more left. We'll quickly go through them and then change topics. So we'll discuss this idea of log analysis. And you may know this already, but logs get generated all the time on every machine that runs. And so this thing called Splunk is a is a service that you may have heard of or no, I don't know. But Microsoft has, if I remember, signed a $5 million per year contract with this company to help them analyze logs. That's the license that Microsoft took from them, if I understand. And what this thing does is helps you analyze data logs from all the machines that you ever have. So what is a log? A log is basically what generates at the end of a run of something. So you have something running on your infrastructure. So for example, here is the terminal, and I have this machine running somewhere. So I will go to connect to that machine. Maybe this one, and I will see that I have some containers running. And so if I want to see the logs for a particular container, I can just extract that log. Let's say if I want to see the logs from, where is that one? A good example would be this one. Yeah. So let's go see Docker logs for that service. So here are the logs showing up. And quite a bit of content is actually dumping right now on the screen. It is really, really impossible for people to digest and understand humans, for them to understand what exactly is going on. Who is visiting the site? What are they doing with it? For example, you can see somewhere here was this Google bot came along. And this bot is basically trying to look at this page called category backup. And that's Google crawling this website. Similarly, you have another bot called uptime robot coming and hitting the site, trying to see if the site is actually available and up and running. Then you have again, Google bot hitting Google bot again. And so you're looking at these things coming and hitting the site somewhere. You have this Flipboard proxy from Flipboard application coming and hitting the site. You have up in here, these are not bots, but these are data accesses where somebody is actually accessing something on a live site. So these are the information that needs to be analyzed. And it is humanly not possible to actually read this log. It is quite long and it accumulates and generates. And eventually, if you don't analyze this, you will find yourself in a situation where you lost interesting insights into your data. And that's where this company comes into play to help you analyze those logs. It's a well-known company, Splunk. Five million dollars per year is a good amount of money coming from Microsoft. That's what they have taken. However, this is a commercial service. So to address that scenario from an open source perspective, there are other solutions out there that basically conceptually put components together to construct pretty much what Splunk does, but in open source. And that thing, the combination is known as log. For the purpose of logging, it's called the ELK stack. And this stands for Elastic Cache, Log Stash, and Kibana. So three different applications, Elastic Cache, Log Stash, and Kibana. Those three work together to construct the ELK stack, which is conceptually very similar to what Splunk does. And they will help you in logging. So let us read a little bit of this ELK stack, what they do. This is Elastic Cache. Oh, sorry. It was Elastic Search, not Elastic Cache. So Elastic Search. Elastic Cache is a different product. So Elastic Search is this open source product, which is what, as you can see, it is using Log Stash. And I made a typing mistake in writing the Log Stash word also. So it is S-T-A-S-H. And so Log Stash, Elastic Search, and Kibana put together to construct a ELK stack is what is extremely popular these days in log analysis. And so what these components do is what Splunk does. And so if you look at each of these components together, the idea behind Elastic Search is to be able to have you query and search and understand and slice and dice the data that gets accumulated. This other product called Log Stash is basically like a, like a, like a, what is that Log Stash there? They got together with Elastic Stash. So this Log Stash is the open source products, man. Log Stash GitHub. That's the way to find it. So here is Log Stash on that location. And if you look at that product, this basically is the parser that parses those logs that you saw in the terminal window. And so this is the primary parser that parses those logs. Elastic Search will help you search. And Kibana is a viewer to view and to report on what you are parsing, but you are actually trying to analyze and understand those logs are telling you. And so these three combinations together are available in a ready-made Docker container. I think it is called Docker Elk. So there should be a Docker Elk stack right there. And you can run this in one line, but basically run it like one line of command. And this is how you get Docker Compose. So basically you run it this one line that will look at this compose file here and construct an Elastic Search, a Log Stash container, a Kibana container and put them in a usable fashion. And you have a ready-made Elk stack in one line. That's what you are able to accomplish if you're using open source without having to spend. And of course, commercial services give you a ton of support with that. So I'm not comparing directly an apples to apples basis, but the concept is if you have this stack put in place for you, you can basically do the same work by putting some extra efforts that are otherwise not available unless you buy commercial services. But you can accomplish the same idea of analyzing your information about the logs that your machines generate all the time. And you saw one quick example. That's what these stacks will do. And that is conceptually what is being illustrated in this diagram where you are basically running same things like I described except, except this illustration is Amazon specific, meaning they will use all the Amazon specific technologies such as S3 and MapReduce, Elastic MapReduce and the CloudFront CDNs. But if you look at it from an objective neutral perspective, you will find that you basically have a bunch of web servers that are enabling your visitors to view whatever your applications are showing. Basically, this is how your users are hitting your web servers. And the web servers are basically generating those logs that you saw in my example, which is here. You saw these logs. So you're looking at them right now. These logs get generated by your machines that you may have out there on some cloud. The idea is to take those logs from those machines and centralize them in some bucket like right here. And then once you have these log files collected, that's the collection part. Everything else, these visitors have nothing to do with your logs. They just visit. They just do whatever they're doing. And so in this example, you have this visitor called Googlebot came and Googlebot looked at whatever it was doing and it went away. So that's the whole purpose of Googlebot is to be able to see what you have in your content. So here Googlebot is looking at this particular style, the CSS file right there. And that's this agent Googlebot comes and sees. That's what this visitor is basically. So Googlebot is one of the visitors. And they come and they hit your servers and then they go away. As a consequence of that visit, these web servers generate a log. And the idea behind log analysis is the whole purpose of web log analysis of this application is to be able to gather those logs and centrally store them. That's the first thing. These visitors have nothing really much to do with the logs themselves. So they just do whatever they're doing and they're gone. They go away. The basic fundamental reason behind you accumulating all these logs in a central location is because you want to analyze them and understand what they are doing, what these people are doing with your boxes. These visitors, what are they doing on the machines? You want to analyze that and you want to basically collect that statistic, the events that happened, the logs that happened. You want to collect them first of all and put them in a bucket. That's the first requirement that you continuously import all the logs and dump it in some location so that you can actually access them later on. Now, having had access to the logs that are continuously flowing in, the next logical concept or logical thing to do is to actually analyze what are these logs actually mean to you. And that is what this ELK stack will do for you. And you can include Hadoop in there. So that's what this illustration does. They have put in an ElasticMapReduce implementation in this big gigantic blob, which is basically analyzing your data stored in the bucket, which contains log files. And that's where this analysis will happen. And the result of that analysis will be stored in a structured database such as RDS, can be PostgreSQL, MySQL, MariaDB. That's where this analytics database will actually store information in a structured format, not NoSQL, but RDS type. So in a structured database with a schema around it so that your business analyst person can actually make sense of what is going on. And that analysis can then view and look at the database itself and try to understand what is going on. So I will give you a quick illustration of what exactly this is what I do, by the way, in our own site. So we have these visitors come and they generate these logs that you see here. And what I do is I run something similar to what you're looking at in the Hadoop cluster in a separate container called analytics. And this is actually showing me analytics of things that are happening right now or has happened over the last couple of days. By the way, this is a fairly recent service that I have just started. And what this service does actually looks at all the visitors that are visiting on a real time map. So it is telling me people who are actually hitting it right now. And so if you open the cloud genius website, right now, you will actually see bubbles popping up right here. So go ahead and hit it. I should be able to see that. So hit the website cloud genius, and we should be able to see the real time map of people visiting like right now. That's what I'm expecting to see a real time visitor, visitor logs, they should pop up. So this was me coming in a while ago. And so I was trying to about three minutes, 49 seconds. And so if you visit the same site, you should be able to see that activity popping up almost a few seconds after you come. And I should be able to get that information that statistics gets accumulated in this visitor log that actually generates beautiful reports for me as to who is visiting. And when do people visit? What do they do? What actions they do? What page titles they see? What entry pages they come from? What exit pages they go out from? What do they download? And things of that nature. Who referred them? Things of that nature, I am analyzing that information. And you can find that, you know, people are coming to this site from this site. Apparently, somebody searching the word cloud genius, the keyword and popping up. Somebody is coming from Facebook, somebody is coming from LinkedIn. So I'm basically analyzing these data in our own deployment of a slightly different view of the same thing, using another open source product called Pwik. And so this product, Pwik is an analysis product. So it's open source. And I'm using this to analyze data about logs that generate like this, that you are seeing on the cloud genius machines and try to understand what the log information is telling me. And that's what this log analytics does from an actual usage perspective. So larger companies may use different solutions. So this Pwik solution is good enough for me for right now. And that's what I have actually constructed in form of a Docker Pwik. So I have a Docker container running this solution. And all I do is run this one line that starts up a Pwik container that does the analysis for me that looks like you saw. And that is what is the analysis part on the right half of the screen. And the business analysis person is me. I am the person that looks at the reports that show up here. And I do have a database, not RDS, but a Postgres database that looks at the data that gets stored after the analysis has happened. This analysis happens inside that Pwik application that you're looking at here that looks like this from a dashboard. So there is an iPhone app for that and an iPad app. And of course, the browser based interface where you can see the analysis that happens for the user, the business user to see and analyze and get insights about what these logs are actually telling you. If I try to read this log myself by hand, it is really, really a nightmarish idea. If I try to read that log, it is humanly impossible. I cannot do it. So I don't even bother doing it. Instead, I will deploy some solution like this, like Pwik, for example, or one of those guys, and understand what the logs are telling me. And so if that is something useful, something meaningful, I can actually take business action around it and make things happen. That's the overall idea behind log analysis. And almost every company I know does it. The best ones that I know actually are Expedia. They are beautifully analyzing logs of about 40 or 50 different domains that they run across the world on people trying to buy airline tickets, trying to buy hotel rooms, basically get hotel accommodations in multiple places worldwide. And the only way for them to analyze what is going on in their business is through analysis of logs. So a couple of people from CloudGenius who actually have found jobs inside the company are doing this exact work in analyzing the traffic that gets to Expedia in terms of making sense of what people are doing. So if you say, for example, go to Expedia.com, and you look at some ticket to say, vacation. So I take a hotel in Maui, for example. So I want to get a hotel here, and I select some dates, and then I search. And so it can give me that output. But then at the same time on the backend, this thing is actually making a log entry on my IP address and trying to figure out. And it's actually right now telling me, as you can see at the bottom, yellow corners, four people are shopping on a Pali hotels on Expedia right now. It's telling me right there, right then and there. And so if I don't do much here and I just walk away, I might get a prompt saying, you know what, looks like you are interested in the Westin Maui resort and for this price of $299. So some engine on the backend is actually looking at my log statistics. And it tells me already that most recent booking for this hotel was less than one day ago. And so it just told me that. I may want to reserve it. And if I do that, it says that 33 other viewers are watching the same room you want to buy quickly. And if I say that, you know what, no, I'm not interested. And if I walk away and if the, if the engine on the backend that is looking at the logs on this site somehow figures out that I may want to walk away on that point, it might actually send me a notification or send me an email if I'm logged in. If I am logged in like this, for example, it might know that who the user is. And in that case, it might actually send me a, so if I log in as a user, I don't know why it's not logging in, but it might send me a notification saying, you know what, instead of $370, come back again and buy this hotel Westin for say $50 off. Here is a coupon for that. Come again. And if I, if I do that, if I just walk away like this and just go away from the site, it might want to analyze my behavior and actually prompt me and entice me to buy the stuff. And that's the only way I think Expedia can figure out to run their business is to understand what these users are doing. The visitors, they're visiting, they're doing something on their websites. They're trying to buy stuff or at least looking for price comparisons or things of that nature, but that is generating these humongous logs that get generated. And these logs are stored in some central location like S3, for example, here. And then they have a huge gigantic Hadoop cluster. And that's is what makes sense that the Hadoop cluster will help them analyze this log data. And these business people inside the company will be able to make sense of the logs pretty much like I described. They have newer methods of log analysis, not using Hadoop cluster, but slightly better, such as this product, Spark, which is a lightning fast solution, probably better than Hadoop on the comparison goes something like, you know, the time it takes to do something is 110 versus 0.9 in Spark. So it is pretty darn fast compared to Hadoop. And it's a newer, better processing technology than Hadoop that does whatever Hadoop does, plus a lot more, much faster, all of it in memory. And this technology is rapidly evolving as well. And this is what is in use. In addition to Hadoop at Expedia, they're using Spark also to do some really quick, almost near real time analysis of what that user may be doing based on the incoming logs from visits of people. And so they are actually thinking of implementing a prototype in which a user may be prompted to who may be interested in booking an entire trip of say $2,000 to go to Maui, a flight to Maui and back to Seattle. So if they decide that and find that $2,000 is too much of a price, they tend to walk away in real time Expedia might prompt them that, you know what, don't go away because there is an exit intention. The exit intention might show up something like this, that you actually take your mouse and hover over to the X button, try to go away from it. That is a way to detect exit intention. The moment you're taking your mouse and go and try to attempt to close it, it might show exit intention. And in that case, it might actually prompt you in JavaScript asking you to not close the window, but here is a $200 coupon that you may want to apply to the $2,000 ticket price that might entice you to buy this, that might entice this visitor to actually buy the $2,000 ticket for a $1,800 price because of the instantaneous or rather near real time coupon that is able to be created based on analysis of logs generated in not like a batch based system, Adobe is a batch based system, but using other newer technologies that are able to generate insights in almost real time and is able to act on that insight, not shown in this illustration, but with Spark, you can enable scenarios like this. And they're actually experimenting with things of that nature to be able to provide insights based on logs and act on them in real time. And that's a thing that is happening, evolving slowly and is put in practice at companies like Expedia. It's happening. You might hear about it at some point. I know the person who's doing it and that's how they just told me. They are actually three years ago finished club genius program. You probably know him or not, but he is on this site. He is here. You may know him. At least you have seen him, this guy. What's his name? Shahul. So somewhere on this list, you will find this guy. That is what he's doing that work. Yeah, this guy. Yeah, not opening properly, but I'm being impatient. I think that's more likely true. So this gentleman, Shahul, is the director of IT at Expedia. This is the person that is actually implementing that. Okay. So this is now a log analysis. If you have any questions on this concept, please ask away anytime. We will now switch over to financial and grid computing, financial services, grid computing. This is the idea that you will see implemented in New York at places where you have a banking system or some kind of a stock based system. So some kind of a corporate data center that involves banking operations. That's the suggested reference architecture that Amazon has provided to us. So we can take a look at what these guys are basically suggesting is to have this illustration called out and actually implemented in your corporate banking data center that you might have, as well as a replica of what you may implement in the context of some cloud. In this example, they're using the Amazon's cloud. So my question to you is, do you really find it interesting for us to discuss it like this or you find that this discussion is boring? If that is the case, I really want to know because I don't want to do, the one thing I don't want to do is to bore you. And so we have three or four more, which are, you know, maybe interesting. Maybe let you pick and choose what you want in terms of whether e-commerce scenarios are interesting. So this is basically how the e-commerce illustrations here, we have part one, part two, and part three is basically what amazon.com is doing. So the commercial e-commerce site amazon.com in three parts is what they have attempted to draw in here. And then here is time series processing, which is a different concept. I'll talk about that as well. And so the question I really have is, is this boring or interesting? Because what I want to be able to also cover is this other idea about security or end-to-end encryption of traffic that goes, which is something that we haven't touched upon. And I think it's a critical element, which is end-to-end encryption of traffic that you may have seen or heard. You talk, people talk about SSL everywhere. People talk about HTTPS. That's what I want to do an exercise with you and make sure that you are comfortable with this idea about encryption of traffic on the internet. It is a security related thing. That's the thing I want to really cover. But at the same time, don't want to bore you. And so just looking at the time we have, we are about halfway point. And the question is, is this boring? That's the honest question. Is this discussion interesting to you or not? Any responses? I can switch straight to the other topic and have you look at these pictures on your own and ask questions if you like that. These pictures do have write-ups given here. And this is the part one of your e-commerce front-end site. The other illustrations we have are part two of the checkout pipeline. This is the idea of what would be your checkout behavior. So you have an item, you put that in the shopping cart, let's say this item, and you check it out. That checkout system is a different site. It's a different service. That's what this illustration does, the checkout service. And then there is the third one, which is marketing and recommendation engine. So you have this other idea behind this site here where people who bought this also bought that and also here frequently bought together. Our customers who bought this also bought those things. That is basically a recommendation engine. So that's what is kicking here, which is marketing and recommendations. That's the third part of e-commerce. And there are simpler, smaller open source solutions also available in this segment that you may want to see if you're interested. Then this time series processing is basically the idea of... Let me first of all describe to you what a time series is. The time series is... Think of it like this. Let's go to finance, that's a good way to describe time series. If you go to time series and you look at the data of a particular stock price, so let's say Apple stock, and you have access to historical data. And so here you can see that the daily price of Apple stock for the last so many years, you can say max. So since 1980 through all the way till yesterday, if you see that I want to get the prices, daily prices on a daily basis, I can actually apply and then download that data. So here is the data that comes along, that stores information that I might need from the Apple stock price since 1980 all the way in a CSV file. So I just have it now. And this is a classic time series, by the way. So here you have date of the item. So at the very bottom, you will find in 1980, the price of Apple stock and the opening price, the high price, low price, close price, volume, and adjusted close price. That's what you're looking at. It was 42 cents in 1980, whereas it is $130 today. So this is a classic example of a time series. And that's the analyzing, the processing of time series data that you may receive in your company of some data that may be created for you. And this is typically sensor data that gets accumulated on a variety of scenarios. I'll talk about that as well. So that's what we are looking at. What I want to be able to do is take a very short break and resume our discussion on the remainder four or five. And then we will switch over to security related topic, which is about end-to-end encryption, about SSL, about this concept that you're looking at in the picture right there. So click NC details, and you see this HTTPS certificate in place for the Amazon site. And many of these sites on the internet these days are actually supporting SSL. Most big companies do it already. Like here, this is Google doing it. This is Microsoft doing it. There are some differences that you may have seen of which is, for example, this Boeing employee credit union doing it. And they have done it slightly differently than most other companies. So here you see this certificate is slightly different from these certificates. And I'll talk more about that after the break. I'll take a short quick break for five minutes. It is 9.32 right now. I will resume at 9.37. And let us finish up those discussions on these architectural diagrams to conclude that list here. And then we'll switch over to security related discussions on SSL certificates. By the way, a quick question. Have you generated SSL certificates yourself ever? Or have you used them ever? Yes, sir. All the time. Yes. Oh, good. Good. Good. Good. Yes. Have you paid money for it? I mean, has your company paid money for it? We had we had our own source servers. Okay. Yeah. For large companies. Yes. And so for other people, there is a new thing available these days that you may or may not have heard of, which is this thing. It's an open authority. So open public authority to generate certificates. And that's what we'll use and discuss deeply. So taking a short break right now. We'll resume. Thank you. Hey, guys, I'm back. Okay, sir. That's right. So let's resume. I'm just collecting my thoughts on time series. Let me close this one back here. Excel crashed. Whatever. So from from an ecommerce implementation standpoint, a couple of things. That this is the reference in reference architecture as suggested by Amazon Web Services for implementation of an ecommerce solution. So from an ecommerce perspective, you primarily have things of things or products or services that you want to sell. And so you basically have a product, a place for it to be checked out. And for people to be able to pay you money for it. That's the idea. So from an implementation perspective, you need some kind of some kind of a product. So this is a product listing P1, P2, P3, like that. Then you have to have a catalog of products that shows some kind of a representation of this thing laid out in a visual fashion where people can add the item into a shopping cart. I want these two items. I want to buy them. So maybe I'll just go look at them first before I actually add to the cart. So for every product, there is also a product page. For every product P1, P2, there is a page dedicated to that product P1 and here is a description, here is a picture. That's the conceptual frame of thinking about what a catalog would be, what a product would look like. And in addition to that catalog listing, you might have a way of categorizing your catalogs in various types of products that you might have. And so you might have, you've seen that in Amazon's example. If you go to Amazon, you have a bunch of categories, all these things. And then you have the, oh, my favorite item right there, this one. And so you can add to the cart. That's a product and I just put that in the cart. So that's what you're looking at, a catalog where you're categorizing in various types and you put products into product pages and you provide an add to cart button. And this button lets people make a list of things that you want to buy. So this is your shopping cart. And that's what this illustration attempts to draw in a architectural view. So if you look at this picture from a top level perspective, you have this user, customer, who is visiting your site. The first thing to do is do a DNS query on your domain name and try to visit the endpoint. The endpoint will very likely be either a cloud front solution, which is where it is providing a front end to either a bucket of some sorts that is storing your static content that sits in your buckets that gets stored and displayed to you here. That's one thing. The other thing that this cloud front will do is to provide you a custom origin, which is this endpoint, which is where you have your e-commerce application running. And this custom origin for this cloud front content delivery network will actually show whatever this application is creating and showing it to you. In this elastic beanstalk, under the hood is basically the same illustration that we have been discussing all along, which is this model of a scalable infrastructure where you have some kind of a load balancer and spreading load across multiple machines, M1, M2, like that. And that's essentially what goes under the hood behind an elastic beanstalk. The word beanstalk comes from the phrase or the story that there is this Jack and Beanstalk or whatever that story is, which grows and shrinks basically. And so the growth may look like this and shrink may look like this. And this happens automatically under the hood in a Beanstalk. And Beanstalk is a service available from Amazon, as you know already, AWS console. And you will go and you will find Beanstalk somewhere here. Where is that? That one here. So you click. And what you do essentially in this example is to let's go to some other region where it is not used. And so in Beanstalk, you basically select a language that you write your applications in. So I'm maybe selecting my Java as an application and I just launch. The moment I launch, I automatically get a scalable infrastructure that looks like this. The whole thing. Automatically receive one. It is scalable. It has a load balancer built in and it scales by itself. That's the service. You can customize it further and say that, you know what, I want this version of Java 7 or Java 8. Or you can specifically say what version you have. Similarly, you can say that I want load balancing and automatically scaling or I don't. In that case, you select a single machine and you can construct your Beanstalk accordingly. And then you click the launch button and then you have a readymade platform that looks like this. And you throw your applications, the code that you will write in Java. You basically send your jar file or your war file, the application itself. You dump it here on these boxes. They will scale for you. That's the concept behind Beanstalk. And so back in this illustration, come on. Back in this illustration, you have this item e-commerce application, which is basically some code that you write in some language that runs in a scalable infrastructure. And that is what this particular Beanstalk application will do is provide you the e-commerce capabilities. That's what a user will see when they visit. Say for example, I have just recently implemented a shopping cart. So this product page that you see is a catalog page where you have these products, 1, 2, 3, 4, 5, 6, like that. People can add this product and you can just click the button to add that item in the cart, the shopping cart that shows up. And then you can check out. That's the idea. Having said, the concept essentially is that this application that runs the e-commerce Beanstalk application runs e-commerce for your company in the elastic Beanstalk solution set also gets visible through CloudFront to the customer. The customer can basically get this application and access the product pages or your particular product set that you might be selling in your catalog and your categories that you might have. In addition to this view of what you might be able to provide to the end user, like this example of Amazon.com is providing this view of all the items in the catalog that you see here. There is also supplementary capability injected into the same page. For example, you will see some marketing happening right in this list, in this page itself. For example, at the bottom, you will see marketing activities. What are the top electronics or what travel necessities you might need or the wishlist that you might have or your browsing history or related items to items you viewed. So they're basically keeping track of whatever you are doing on the site and are able to provide you recommendations on this cookies and cream macadamia. And based on this information, they're also able to say customers who bought this item also bought those items. And so this is basically coming from this other engine, which is in the black color elastic beanstalk right there, which is this one is running your marketing and recommendation service. It's a separate scalable stack that runs the application that generates those recommendations. So this is a recommendation coming from Amazon's recommendation engine that gets coupled with the main page generated by this beanstalk, which is the e-commerce application itself. And this site will generate the page content. This recommendation service injects its recommendation pretty much like this. So anywhere below this item, all the way down up to this segment and some sponsored related products to this item. These are sponsored as in advertisements. And then you have special offers. And those three elements, I would say, special offers, sponsored products, and customers who bought this also bought that. Those three segments are actually getting constructed by the recommendation service and the marketing service. It's a separate elastic beanstalk injecting its components inside the main page that the customer sees when the customer is trying to buy something from the e-commerce application. This recommendation will get injected right there. Similarly, on the other hand, on the right side of the screen, you see this page here, which is showing you the cart. This cart component also gets injected into the same view by the checkout service. So it's a separate service running in a checkout application, which is a different elastic beanstalk. It's a separate presence. And very, very important is this padlock, which is secure connection. So you are basically making sure that any financial transactions that happen in the checkout process are HTTPS secure. Whereas this is not necessarily need to be secure, although it is highly recommended that the entire thing that you do is secure HTTPS end to end. And that is the way to go forward. Any services that you run or that you have your customers touching and accessing, make sure that you are putting them in a secure protocol like HTTPS, for example. And in this picture, the picture doesn't actually indicate that it also needs to have a HTTPS protocol built in right there. They just don't do that. But I think it's a bad idea. You should have secure connection everywhere, not just for your checkout pieces. Now, having said this checkout application is actually injecting itself into the front page, like a shopping cart element. So you see this whole block on the right side, as well as the ability to add an item in the cart for every page. So you go to this page, and you have this subscribe now button. That's a shopping cart method. So you just click the subscribe now button. For example, it will take you to the shopping cart application. So you're basically transferred over to this checkout service right now. Now, these are the three main applications you are seeing here, e-commerce application, checkout application, and recommendations. These three applications themselves are also producing a ton of data. And what they don't show here, don't show you here, is the ability to collect and generate logs, which is probably a separate application set. However, these applications, primarily the e-commerce application, is looking at your database of products that you store somewhere in a data store. For example, they are showing you here, product catalog, and the sessions that people maintain. This customer is going to maintain a session with you. And that session is persistent, meaning you might want to walk away from this and close the window, and then come back again on a different computer and log in. And you will see the same exact item in your shopping cart, just like it was on the other computer. They're able to do that persistence of sessions by storing the state of the user in the database at the very back end. They don't store state in the Beanstalk, in the stack itself, not here, but here. The persistent of user's session is stored somewhere away in a database. So that this user, when they use one computer, they see the item, they saw it in this shopping cart, they go away, use a different computer, they log in, and they see the same item in the shopping cart, primarily because of the same state right there. And that's what this illustration of DynamoDB store, which stores your product catalog and user states. They also have a search engine here, for example, that they are showing you to use Cloud Search. But you can use any other search implementation, such as Elasticsearch, for example, which can also basically search your catalog and present this e-commerce application with a view that the application may want to show to the end user, like this. So when I search something, and I say I want to buy an iPad, so here is the result that I see. The result came back because of a search query that this user executed that ran on this application, which basically passed on that request to the search engine that searched the product catalog and brought down a subset of products that matches the search query, which is, in this case, the iPad. And now you're looking at a subset of the entire catalog that focuses just on the iPad based on a search that this query ran on the database and is able to show to the user through the e-commerce application. And the customer is now looking at a result that is the subset that they wanted to search, the iPad, for example. That's the ability to be able to search what you have from your catalog. Similarly, they're using Elastic Cache. And this cache is the idea that I think I mentioned a little bit ago. A couple of days ago, I mentioned that I was using a caching system to store database queries that are repeated. Repeated queries are stored in a transient session store or a catalog cache. What does that mean? I'd like to explain that a little bit, what this thing is. So if I come here and then close the window, open that window again, go to Amazon, and another user might, just like me, another user might also search for iPad. Click. And another user might also come again and search for the same word iPad again. And so it doesn't make sense for this search engine to search the database every time somebody is searching the word iPad for the product catalog. It is a waste of time for this search engine to actually keep searching that data catalog again and again and again. Every time somebody comes here and says, show me all the iPads you have please. So here is go iPad one more time. I mean, it is really bogging down the database if you do this over and over. So to address that problem, and you don't want to bog your databases, you don't want to waste time in searching the results over and over, that you actually cache the queries that are repeatedly executed. So whatever catalog queries are getting run over and over in Formos for Search, that you first time do a catalog search, for subsequent searches, you don't. Instead, you store that result in a cache. That is what this Elastic Cache does from Amazon. Elastic Cache is a service that Amazon basically implements using an open source product called Memcache. And so this is a distributed, yeah, read that, Memcache is a free and open source, high performance, distributed memory object caching system, blah, blah, blah. What this does is it caches results from a search in a database. So basically response to a database query. Something like you have a query on your database. So say, no, no, no, not here. So you have something like maybe this. Yeah, so here is a database and I may be able to connect to it, or maybe not. I think I need to connect to a database to show, but you know, you understand a query. A query is something like, let me write it for you. Select everything, or rather, select iPad from catalog. Something like this, right? I want to search the word iPad from the catalog. And then this is the result that I will acquire in form of a JSON structure that comes out from the database that I will make use of. So here is iPad number one, iPad number two, like that. The results are popping up. And this result of a query that I need to store and reuse over and over every time somebody is searching for a catalog for the word iPad, I should return this result. Every time you search here, search again here. And this element needs to be cached. And that's what this memcached will do for you, which is this application, which is open source. There are also this application which also does the same thing. Redis is also another cache. It is also an in memory data store that is used for databases, caches and message broker. You can use it like a queue, you can use it like a cache, or you can use it like a database itself. Three uses of the same product, Redis, open source. And this is a very similar product, memcached. And that is what is implemented in this Elastic Cache product from Amazon. And that is here. So somewhere in this listing, you will find a product called Elastic Cache. Let's go find it. Elastic Cache. Let's go search for it. Yeah, there. So it's a database. In the listing of databases, that's what you will find, Elastic Cache. And what is it? It's an in memory cache. So it's a web service that makes it easier to launch, easier to manage and scale a distributed in memory cache in the cloud. Basically, it is memcached. That's what it is. You can run it yourself. You don't need to spend money on their service if you know how to run memcached yourself. And it is fairly easy to run memcached yourself. Or use Redis, which is also easy. And both of them come with a ready to use Docker container. So you can just one line and run it. That's what this thing does, basically. Elastic Cache. That's what this thing is running right there. And what it is doing is serving results from a query that was generated some time ago. And I just repeat the result like this. Okay. Okay. Without actually executing the query all the way through the catalog. That's what caching does from a database perspective. So this search engine doesn't have to hit the backend over and over again. It will just give this application the result from the cache. And the result is shown to the consumer to the end user quickly. That's the overall front end elements from a design perspective from an e-commerce site. The web front end parts. In another diagram, like has shown in by Amazon, this is part two, which is focusing on the checkout elements. So the same picture that you saw before on the right side of that picture will be this checkout application. The e-commerce application is this now black box, which is the web front end. The elastic beanstalk that is running your front end. That is the e-commerce application. And on the right of it is this beanstalk, which is basically the checkout application. Now inside the checkout application, you have the customer who is interfacing with the checkout application that looks something like this. You've done it before many times. This is what it looks like. You check out and you want to buy some stuff. And then you want to buy this next day delivery. And so you pay more. And maybe you don't want to buy this. You say, no, I already have it. Don't want to buy this item. So I just want to buy the chocolate and I will receive it on Monday. And that's great. And so I place the order or maybe Tuesday for a free shipping. And then I place that order. That's the idea behind an e-commerce scenario. And what this application does is exactly what the user will be doing with the shopping cart. What does that boil down to is that you have a scalable infrastructure, which is basically now looking at a structured database like RDS, not Dynamo, not Mongo, but Postgres or MySQL structured databases, MariaDB type databases, where you have a schema to begin with. Give me a second. Hold on. Where you have a schema to handle. The schema is at the time of writing as opposed to schema at the time of reading. And we discussed this last time briefly on schema on read versus schema on write, SQL structured databases versus NoSQL databases like Mongo. So this database RDS is a structured database like Microsoft SQL, like Oracle database, like Postgres, MySQL, those types, which is the dominant usage of a database when you are dealing with money. So this checkout application is basically going to take money from this customer and then create an order. That order needs to be sitting in some kind of a workflow. And that workflow, which is a service that you need to construct, basically something like this, will look at your customer's orders that they will send. The customer will place an order through the checkout application and the order will be received and stored in some database so that you will have a very clear idea about what you are buying. So you can go back to the site and look at your history of all the orders you purchased. That's what this database is actually storing the orders inside RDS database through the checkout application. Now on the other end, on the backend of this database, is the ability for your company to actually deliver against the order that came in. And so you have this customer that put in an order, then you need to notify that customer immediately that, you know what, you have ordered, so I'm going to thank you for your order. Yeah, we got that email for you out. So as soon as you order something, you will receive an email through the simple email service that will send you an email saying, yeah, we got the order. We are going to give it to you, send it to you in the next two days, three days, whatever it takes. So maybe right now in a drone delivery system, any of those systems that you will put in place for the order to be fulfilled, you need to now worry about the fulfillment, which is where these workflows come into play. And so in your warehouse, where you are storing your stuff in your inventories, you have mobile workers that are basically getting notified of the order that you receive in your warehouse. And so your people that may be handling, but I think in case of Amazon, it is completely automated where they don't have people, too many people, at least maybe one or two people. I've seen some videos on it, internal videos from Amazon where the warehouses do not have people at all, or maybe one or two that are just looking at what's going on. And most of these machines are actually handling customers orders and actually making sure that these boxes that get put together and be sent to you via UPS, FedEx, local shipments. And these days, Uber, even Uber is delivering customer packets on a same day, next day delivery scenarios. So Uber people are taking items from the warehouse and delivering it to the end user customer. As an extreme example, in certain parts of the US, it's happening right now. So having the backend on your warehouse where either people or some machines, have you seen that video at Amazon warehouse? They show you Amazon warehouse YouTube videos behind the scenes of an Amazon warehouse. You should watch this video if you haven't watched it. So you should see that video if you haven't watched it, and I'll paste a link for you in Snapchat. And the idea essentially is that these workers, which may be people in some companies, it may be machines in some other companies, are basically looking at the customer orders and making sure that they are fulfilled. That's the entire other backend processing which doesn't necessarily involve machines, but it may involve people. For other companies, it might involve people. For Amazon, it is all automated. And that has got the fulfillment part, checkout fulfillment part. There is also a third segment back in here, which is the marketing and recommendations part. And that involves analysis of similarity, of profiling, of things of the nature that people like you who live in that area may be interested in this type of item may also be interested in the other type of item. And that's the recommendations engine that you may have heard of in the case of Netflix discussion we discussed briefly about recommendations. People who watch this movie also watch that movie. And that's the way they are basically keeping you glued to the Netflix service, for example. Same concept here in the context of e-commerce is that people who bought this item also bought the other items that you may be interested in. So here is a list. And that is a classic recommendations engine. This concept of recommendations engine is now open source. Open source recommendation engine is also available. And so here is a long list of open source recommendation engines, apparently. And so we have a bunch of them that you should take a look if you're interested in that direction. In addition to this list, there are some ready to use components that you can put in your application. So if you're writing some application for e-commerce, you can actually use ready to use sources such as this service, Prediction.io. It is a part of an Apache incubation project. So if you look at the top level URL, it is apache.org. So Prediction.io is a real time recommendation service now going to be a part of Apache. Right now it is in incubation stage, which means it will eventually become, if it succeeds, it becomes a first class Apache project. And that's what this will eventually take shape of. But as you can see, it is incubator right now. Once it comes out of incubator, it becomes a first class project on Apache foundation site. And at that time, you can just see all these projects that are there. And Hadoop is one of them. Spark is another bunch of other projects. All these projects are in Apache. And eventually you will have a recommendation engine from Prediction.io showing up right in this list. Let's see what you have recommendations. Not in the main top level list, but you do have it in your incubation. So that's what this you're looking at. You can very simply integrate your application just like they describe in this page. Have that application collect and analyze data. And so you can basically, if you are into developing this thing, you can use this open source project and construct your own recommendations engine based on some infrastructure that actually you will actually run in a analysis stack. That analysis can take shape of a batch process through Elastic MapReduce, which is basically this gigantic box, which is a Hadoop cluster. Or you can use other methods, modern newer methods that use newer technologies such as Spark to enable your analysis in real time. And then give that recommendations back to the user, this end user that is actually interfacing through you through the other application, which is the eCommerce application, the front end parts. This user is your marketing manager who's looking at the marketing application, basically making tweaks, modifying and understanding what the recommendations are coming out from the MapReduce or Hadoop cluster, whatever the Hadoop cluster is telling you. A person who is internal to your company, who is understanding or trying to understand what that analysis means and making some adjustments, helping write marketing communication in form of emails that go out to people who receive your marketing emails. These customers might receive them and they might come back to buy more stuff from you on your front end. And that might happen based on the service that you send in form of an email to the customers that the marketing manager decides to write through automation, these emails. Alternatively, the orders that you receive will also get analyzed here in the Elastic MapReduce database. And your recommendations service that you will have will also help your front end user on the front end of your site with on the Spark real-time recommendations that get embedded into the front end, like shown in this picture. You will also see the same recommendation service on the other chart that if you go back here, you will see that the same marketing recommendation service is actually plugging itself into the e-commerce application as shown in the part one of the diagram. The same thing shows up again here. And that is the part one showing here. And rest of the picture on the upper left corner is the marketing as well as recommendations engine, which is basically driven by customer's orders. And so whatever customers are sending orders, you are now generating recommendations and then giving those recommendations in an automated fashion. And also a person writing emails and sending out marketing communication so that people will want to visit later on. They want to come back again to the front end and buy more stuff. That's the overall e-commerce illustration that Amazon has put together for us to understand. One thing I would like you to remember, these illustrations might show that they are using Amazon services, but bear in mind that absolutely nothing prevents you from using the same concept in any other cloud or any other scenario which doesn't even involve a cloud, as long as you have some machines that will operate just like the way you want. You can construct same concepts in any location. It doesn't have to be Amazon. It is nothing that is specific to Amazon that you cannot do elsewhere. But the fact that they have put together these illustrations and these charts and diagrams are actually very nice for us to quickly discuss and understand what is going on for a use case. So we talked about time series briefly. And the idea for time series basically was that you have some kind of a variable and it has a value. Let me discuss the value of stock price. It is $113 yesterday. And back in time, the value of this price was 0.04 something like that in 1980. That was the value of the stock in this time frame. The same item at a different price back in history, back in time. And so you have basically values for this variable and a value for that variable in a time series. That's what a time series essentially is. And so time series has multiple use cases. So this is just one example of a stock price. Stock price in time. But time series has a bunch of other uses like a huge lot of them. So many of you know that there is a gas pipeline that travels from Canada and it goes through our state, the Washington state. The gas pipeline is, I think, I forget the name of the company, but it is pretty much goes through east side. That's what supplies our gas to our area, our neighborhood where the east side people live, maybe parts of Seattle also. But this huge pipe that comes from Canada goes all the way south somewhere is traveling through our area. And so this pipeline that travels gas pipe is a critical resource. And so the company that manages this pipe, I think it's Williams Communications. Williams. Williams Company. Williams Communications Company manages the gas pipe. And that company is actually, it has deployed sensors. Sensors to collect data about flow of gas. How the gas is flowing from there. I don't even know whether it flows this way or that way. I have no idea. But it flows somewhere. And it's going and coming from somewhere to somewhere. But the flow of gas and how smooth, how laminar the flow is, or how turbulent the flow is, and what's the temperature, what's the pressure, what ambience in the neighborhood in that area surrounding the pipe. All these are sensor outputs. And they are continuously collecting that data for the last so many years. And this is another example of a sensor data that fits into a time series. Now, if you understood this idea of time series, it has tremendous applications in a variety of businesses. So if you go look at the diagram here, in this example of time series processing, what you are really looking at is the example of Williams Communications Company. The gas pipeline company is collecting sensor data, sample data from a variety of sources, such as the pressure, the flow, the temperature, the ambience around the pipe, wherever the gas is flowing. And it has been doing this for the last so many years ever since the pipeline got created. And that data continuously gets acquired through this SCADA system. SCADA is Supervisory Control and Data Acquisition. That's the system that most companies will deploy for measuring through their instrumentation and collect information about what is relevant to them. And that SCADA will actually access and collect sensor data for whatever they may be measuring. Like in the example of your gas pipeline between Canada and the US, the Williams Communication Company will be measuring and collecting sensor statistics over a period of years and years on an hourly basis, or maybe even more frequent than hourly. And that generates a ton of data already. And that data needs to be stored somewhere and analyzed. And that's what this picture is attempting to illustrate from a conceptual perspective. Again, it doesn't apply to a specific example of Williams Communications Gas Pipeline sensor data, but the framework is what we are looking at. So from a remote sensor here, remote sensor messages, they are arriving, they will come wherever they are coming from. So basically these sensors are whatever the sensors are generating and sending those are basically messages. These messages come in to a queue and just accumulate. The accumulation is basically conceptually similar to people standing in line to get a driver license at the Department of Licensing. And that's a queue. This queue contains messages from remote locations, wherever the pipe is, from Canada all the way down south. And everywhere along the pipe, every hour or every so frequently than an hour, you will have data about what is going on with the pipe. And that data is being sent to your infrastructure. And that is what is accumulating in a queue. That sits in a queue service. Now they have used the example of simple queue service, but you can use any form of queue. You can use Redis as an application to construct a queue, for example, or use RabbitMQ or other queuing techniques and that queue will form and sit right here and get piled up. To process that queue, you need some kind of autoscaling worker machines that will automatically scale and work on the queue and basically involves processing this queue and making sure that the sensor data that comes in gets handled appropriately and is stored in a database systematically. And then you have your SCADA system that actually makes sense of your incoming data that you store in a central location in a DynamoDB, for example. You acquire the data through SCADA and do analysis of whatever your corporate people want to do with the data. And you may have some custom applications to understand gas characteristics in terms of how the flow is happening, if there are any trouble spots. And since you're collecting this data continuously over a period of last so many years, you have a ton of information that can be used to identify and preempt any potential problems that might occur in case of something going wrong with the gas pipe. Like for example, a leakage or something like a fire or some other disaster that might happen across the gas pipe. You can very nicely detect potential budding problems ahead of time by the fact that you have data corresponding to characteristics of various metrics that get collected across the pipe for the last so many years since the pipe is in place, because these sensors are actually generating that data and that you're systematically gathering them. Now, your custom application can actually detect anomaly and find out what may be wrong at a given point. So you might have a pipe, the pipe flows like this. And at some point, you might see that something is going wrong here, the temperature seems to be shooting up for some reason, and you detect that. And if it reaches some threshold and it crosses that threshold, you can actually have a person study what's going on and maybe send that person over to that location and inspect what is wrong? Why is the temperature higher than normal? Over the past several years, we know in the month of October come Halloween time, the temperatures are like this, but this year temperature is different. Wonder why? And that analysis can be done by this person by studying the data themselves or by sending a worker onto the site and identifying and see if there is something wrong in that spot. Because you now have a systematic method of studying the entire pipeline by putting a systematic method to study your data that is basically time series. And you will collect that sense of data, analyze it, and also do further processing if you need to, and then systematically store that data in a Redshift. Redshift is a classic service from Amazon, which is Redshift. But it is basically your data warehouse. So think of Redshift is like Teradata, if you have heard of this company, Teradata. So Teradata is a data warehouse solution. Another alternative solution available commercially is Oracle database that is also used for database data warehousing applications. Another good one is MS SQL. This is your SQL server 2016 is also very well known data store. And so these are software applications, by the way. So Microsoft solution, Oracle solution, as well as Teradata solution. These are database solutions, whereas this thing is a service. And so the idea behind software like this and this, and a service like this is different. The idea of a service is that you don't have to manage any software, it is just available to you. You pay 25 cents an hour. And that's what you get. The pricing is fairly simple. Start small for 25 cents per hour with no commitment. That's the pricing they have mentioned here, start small, 25 cents per hour. So about $200 a month, you can get a highly scalable database service available to you for long term retention, and is systematically structured like a data warehouse for your company. You can actually start using this service if you are interested. However, it depends on the company's business, whether they are interested in whether they're reinvested in a Teradata solution or Oracle solution or a Microsoft solution, they may be having one of these solutions already in house. In that case, of course, you can use your in the house solution, but Amazon has put together this service, which under the hood uses Postgres SQL. That's the underlying thing that is used under the hood by Redshift. It is actually compatible with a Postgres implementation you might have in house. So this is an open source product, as you know already, Postgres SQL is what is used to build this Redshift solution. And that's what you're looking at. In this example, you will store your entire data about historical time series of all the samples you will ever collect for your gas pipeline, analyze it, take action on it, and provide that insight after analyzing through a Hadoop cluster into your Redshift so that you can have this business user look at and do queries on directly on the Redshift without having to worry about raw sensor data, because the sensor data can be really cryptic. It is hard to understand for a business user. So they would want to access this through a systematic structured database store, such as a data warehouse like Redshift, for example. That's the illustration you have in this picture. That finishes this short list. There are more, by the way. There are more. And we can keep discussing these things if it is of your interest. If you have something you want to discuss that is not in this list, and you still want to talk about, please bring it up. Because I will cover that. If you are interested, and if you tell me, I will do that. What I want to do next is to maybe pause for a few minutes and ask you if you have any questions, and then change over the topic of ensuring end-to-end security through this method, which is a relatively new thing called Let's Encrypt. And it is open. It is an open authority. So it is available for the community, for everybody. And it is free, of course, and automated. The fact that it is automated and open makes it really, really interesting. It is backed by Linux Foundation. It is actually a product of the Linux Foundation. And so that's the reason why I personally like it. And I use it. You can see that I'm using it right here. That this certificate you see here is actually provided to me by Let's Encrypt Authority. And so this happens through a Docker container. We'll talk about that. But before we jump into this discussion, I want to take a few minutes and ask you if you have any thoughts, any questions of what we have discussed up until now in a variety of architectural illustrations that Amazon has put together for us. And just I want to hear from you a brief before we switch topics. No comments? No questions. But each of the architectures have given understanding of different kinds of scenarios, cases where specific architecture has to be adopted. I think that's the reason why I actually want to cover that. Because it gives you a flavor of various industries, a variety of use cases, how people may want to use the cloud for what they want to be able to do. So that's the reason I actually go through those to give you a flavor. Because various industries do things differently. And various use cases are different. And one thing that you will notice is that for almost every single thing that you want to do today, there is an open source project to help you. And that is something that is phenomenal. And I want you to take note of this, that it is the community of people like you and me and everybody else together are constructing projects on GitHub that help us accomplish whatever we want, whatever we come up with, whatever ideas that we have today. In fact, I will argue that this this entire suite of collections, most of it, by the way, is actually derived from some sort of open source. So you go here. And you start looking what they have. You have EC2, which is a machine service or hyper machine hypervisor service. It is based on the project called Zen hypervisor. This project, which is an open source project, the underlying hypervisor is Zen from Amazon. This is the project, the Zen project. If you now look at the other service container service, the underlying project is Docker, which is also an open source project, Docker containers. If you look at Beanstalk, it is auto scaling, but it uses a bunch of open source technologies. And even dotnet is open source today. It was not until recently, but then even dotnet is open source. So all these open source technologies are available for you to auto scale. This is a scaling service, by the way. Back in here, let's look at Lambda. Lambda is something that I also want to talk about. This is a new service that has been released by Amazon recently. And that actually is fairly recent. The concept is similar to what you saw when you used an S3 bucket and you created a site without server. You've done that before, right? A site without server, static site. And so when you don't have a server, there is nothing to manage. I mean, it's so easy. The same idea, but this is storage, right? The same idea when you can have a full blown complete dynamic site without a server. And that is what is new about this Lambda business. Lambda will help you construct a dynamic site without a server. That is what this thing is here, Lambda. And if you read the top line, it says, you give us code, we'll make it run. Don't worry about the server. That's what this basically does. To compete with them, Google has created... What's the name of this thing called? Google Cloud Functions. The name is called Google Cloud Functions. This is an alpha release just coming out. And you may not have access to it, but I just have got access recently. They have given me access to this alpha code. It is called Google Cloud Functions. And this is to compete with Lambda. The idea here is that all you do is give them code and you forget because they will decide what needs to run, how it runs. There is no need for you to manage a server or a bunch of servers because it is basically conceptually similar to what you do when you take a bucket, convert it into a site. You're basically using storage, but without a server. You're running a static site. Same exact idea, but for dynamic sites. And yes, don't need a server because they'll do it for you. What you do is that you go to Google Cloud Functions. Let's see if I... I think I'm still getting access enabled. Let's see if they have enabled access for me. The last time I checked, I did not have access, but let's see if it is enabled. So Google Cloud Functions shows up there. And okay, now I have access. So what does that boil down to? Google Cloud Functions, I will have to create a function. And that basically means I have to upload some code. And so I can create a function and give it some name, allocate some memory. And by the way, this is just a memory allocation. You're not talking about machine allocation. And you will upload a zip file and then you define the initial function to execute. This is the main program or main.c or main.py, whatever you want to invoke and you create. That's it. That's all it takes to run and get a service, a dynamic service running using Google Cloud Functions. That is a alpha product. Google Cloud Functions, like Lambda, supports three languages right now, not too many. So right now these guys, if you get started, you can construct, I think a Python function, a Java function, and a JavaScript function. There are three different functions you can configure. So you write code in three languages. That's what is supported right now, as I understand. So what it means from a usage perspective is in this Lambda scenario, you give them either Python code or Java code or JavaScript code and give them a starting point, like a main point to begin from, the main function, and upload a package containing your code. That's it. They will run that thing for you through the Lambda service. That's a new development coming out from both companies, Amazon as well as Google. I think Microsoft is, I don't know what Microsoft is doing. Maybe you know about something like Lambda, but I have no idea. And that's what this Lambda is. If you go scroll down, you have this, it's a service called Server Migration. So this service basically helps you migrate your on-premises servers over to a region not supported, so you can select a different region. And then you can take your on-premises servers and migrate them over to Amazon. That's basically like a forklift operation. You take your servers that you have on the campus, they will migrate it for you. That's a service. If you scroll down, under the hood, you will find Elastic File System, which is based on this technology known as Gluster File System. Gluster FS is the cluster file system for your cloud, which is an open-source product. If you keep scrolling down, you will find RDS is also a bunch of open-source products. You will find that it supports Amazon Aurora, which is MySQL-like. I don't think it is open-source. This one is not, but it is basically compatible with MySQL. This one is open-source. This one also is open-source. This is also open-source. This is not, and this is not. You have a bunch of open-source technologies under the hood that are being used. If you scroll down like this, you have DynamoDB, which is a derivative broadly of MongoDB. They have customized it a little bit, but you can practically run Mongo applications in there with some caveats. Elastic Cache is basically Memcached, a variation of that. Redshift is based on Postgres SQL. DMS is a migration service to migrate your data. VPC, the concept you already know, and we have a VPC exercise through automation coming up next time. If you keep scrolling down, this is Hadoop implementation. This is Elastic Search service, which is based on Elastic Search, which is an open-source product. Machine learning also is a capability that uses a bunch of open-source technology. If you scroll down item by item, you will find bits and pieces of open-source used under the hood. That's the key thing I want you to remember is that people... There is, okay, serverless architecture functions, so Microsoft also has Azure functions right there. It's in preview mode, I will try them eventually. Here, you can probably run a language like C-sharp or JavaScript, two choices, and you create a function and basically invokes the same idea. We will have... Eventually, I will construct some examples that I will include that will help us understand how to use Lambda, how to use these functions in Azure functions, or Google functions, or Cloud and that's what you will see inside this Cloud Genius website. I will eventually have some examples for us to play with. There is a pre-made function, by the way, so let's see. We quickly start with the pre-made function. We want to see a timer function in JavaScript and create with a Microsoft account. Let's see what happens. I'm just trying it first time and David just gave me a link to it, so I'm just invoking to see what happens. Just a few more seconds and it takes us to the Azure Cloud and it's going to run some timer function for us. Let's see what they have done. This is my first time experiencing Azure functions. Hang on while we put the fun in functions. Good marketing. There we have. It's a free trial they've given us and it is still loading some things. Here is a function and I want to run it, so I just run. I want to write something first. I want to save it first and then run it and here it says, welcome. You are now connected to the log streaming service. Function started. Node.js timer trigger. Function ran. Function completed successfully. 202. Function was accepted. That was the my timer function and it ran for us in JavaScript as you saw in this log output. You run it again and it runs and it runs again and it runs and you have the log accumulating at the bottom and the function seems to be running without the need for a server. This is running JavaScript code in the browser. Actually, probably it is running on the backend in the Azure cloud, not just here in the browser. You can see the files are located. There are these three files, JSON function, which is this function itself, index.js, which is the invocation, and some readme to help you read and understand what's going on. Here is the readme documentation to go along with that. I think it was a good idea. Thank you. Thank you, David. David just gave us that link, which is Azure implementation of the concept of Lambda, like Google Cloud Functions or like this Lambda business. That's what we have. Now, let us talk about SSL. We have only 15 minutes. Wow. The concept of SSL is basically end-to-end encryption of traffic. You have user. You need to give them warm and fuzzy. That's the core idea, warm, fuzzy feeling. How do you give them warm and fuzzy feeling? If they're using a web browser, you give them a green-colored padlock saying it is secure, something like that. It's all secure. That's the communication you want to send to the user to make them feel warm and fuzzy about using your services, showing you a padlock that your transportation of traffic from what you are sending to the sites are end-to-end encrypted. That fuzzy thing shows up here in form of this padlock that you look at that looks like this here. That is an indication of a secure site. The reason browsers are able to show you that it is actually secure is because of a couple of things. Those things are at the very foundation of every operating system, every operating system, whether it is Windows or Linux or Mac or iOS or Android, any of those, basically any. All of them will carry what is known as a generally accepted list of two servers. These are by default trusted and generally agreed upon that we trust you, we trust you, these servers are trusted. The trust is established at the root level, at the level of operating system. We trust Microsoft to give us Windows. We trust Linux to give us Linux. We trust Apple for Mac and iOS and Google for Android. We trust these companies to begin with. These companies collaborate to put together a list of trusted root servers inside the operating system right there. What does that boil down to? If you go take a look at a machine and let's see if we can connect to a machine. Here in this machine that we are connected to, we should go and find out if we have a list of root servers. Let me see if I can find it. Now I am in this location. It's called HCE SSL certs. And in that location, I have a bunch of servers with their certificates stored in that location already. And it's a huge list by the way. It goes on and on. This is the list of root certificates already placed on an operating system that you're looking at. You will find some very well-known names here in this listing. And some not so well-known names. One of the famous names you will have is this RSA security, for example. Other people that you have here are GeoTrust, GlobalSign. Some other ones are AdTrust, Equifax, and Trust. And like that. So DG cert, another company. Deutsche Telekom. Like that you will have all these companies, Visa, GoSign China. So a bunch of companies are putting their certificates at the root, known as root servers. They are found in this location, HCE SSL certs. On every Linux operating system you will find them here. Here are the root certificates. In other operating systems you will find them in different places. For example, in Apple, you will find it in, what's the name? Keychain. That's where the certificates are stored somewhere here. And so the system roots, that's where the roots are stored. And these are all the certificates stored. Basically, the same list that you saw in Linux, you're seeing it look like the same list in Apple. Sometimes you have some special certificates added on by either a user or by the company that wants to trust somebody else. For example, here may be this certificate that you're looking at, this vrk.gov.root certificate. So it's something different that Linux doesn't have. Or maybe they do. vrk, I don't find it here. So vrk government was not there. vrk.gov.root certificate was not there. Visa information delivery was there. Visa e-commerce root was there. But certain companies are different among operating system vendors. You will have different lists. But generally accepted is a list of root servers that everybody carries. And that is the foundation of all this mutual trust that it establishes. What does that mean? It means this user who's using one of these operating systems and a browser in there will actually see a little lock symbol, primarily because on that site that he or she is visiting www.whatever.com, that the machine on the backside is actually sending you a certificate that matches the root that you have. And if that's the case, this browser will show you green colored item. If it doesn't match for some reason, then you'll see a red item. And you'll say HTTPS mismatch. And that's how you are basically giving this user a warm and fuzzy. Now, in order for us, as a new entity on the internet, if you want to receive a certificate that you want to place it here on your servers, and then you send that certificate to an end user, and you are fairly new. So nobody knows you, nobody trusts you, and you don't know anybody in the root companies to actually allow you a relationship with a trusted root certificate. In that case, what you need to be able to do is to follow this concept of Kevin Bacon. You know this concept of Bacon number. I know you know him and he knows Kevin Bacon, so my Bacon number is three. You know this concept, right? This is called transitive trust. So basically, I know you and you know Mark, and Mark knows Kevin. So therefore, I kind of sort of know like this. So this is me. I know David, David knows Mark, Mark knows Kevin, Kevin Bacon. And so I can eventually transitively say that I know Kevin Bacon through this relationship. And this chain of trust, I trust David, David trusts Mark, Mark trusts Kevin, and therefore, transitively, I trust Kevin. That's the trust relationship I establish. And one can take this chain of trust. That's the concept that you will see employed in when you get an SSL certificate. A certificate is obtained primarily for some kind of a name entity, a DNS entity, for example, like www.whatever.com entity. That's the primary use case. There are many other use cases on certificates, but we are discussing this concept. And so we have to obtain a trusted certificate from somewhere. And we can get straight from the root. So we can go straight to the root authorities and ask them for a certificate. But then, you know, asking Deutsche Telekom, for example, or maybe maybe one of these companies, I don't even know where they are. And so it is really hard for me to approach them. So in order for me to get a trusted certificate, I need to maybe go to an intermediary who might have an intermediate authority. And these are root authorities, by the way. So I might be able to go to an intermediate authority who might have another interim authority who might have another interim authority. And through this interim authority, maybe I can get a certificate which in turn is trusted by this, and this is trusted by this, and this is trusted by this. And therefore I have this transitive trust relationship all the way to the root. And once I have that end-to-end chain of trust established, then I can prove to this user that my site, that I have this new site called www.whatever.com site, that I have this certificate that I'm going to show you, is actually a trusted one because it has a chain of trust all the way to the root that you seem to carry in your hand. And therefore it shows you a green colored lock symbol, which gives you the warm and fuzzy. That's what it basically boils down to. Now in order for us as a new entity on the internet to get a certificate, the classic typical example was to buy a SSL certificate. So you can buy a cheap one for this company, from this company, and you can get a comodo certificate for $76 a year. And that's okay, but this is the boring old way of doing things. I don't want to go this method because first of all I have to go and buy and then go through the motion of going through whatever they are making me do. I don't want any of this because I find it actually silly to go through this route. Most companies, by the way, today do it this way. Most companies do it this way, except the very big ones who have their own authority like Microsoft, for example, or Google, they are their own root authorities. So they don't have to go through this route, but most other companies will go through some of these routes. Very well known is VeriSign. VeriSign will give you SSL, very well known and very expensive. You try to get a SSL certificate from Symantec VeriSign. They will charge you an arm and a leg, $399. Why? Why do I have to do this? When there is an alternative available? The alternative, as you know, is this. And it is a fairly recent thing. And that is going to be the dominant way going forward, at least for web services, because it is automated. Automated. What does that mean? It basically means like this, something like this, that if you, without automation, you have to actually go to this company and buy them, pay them money like the $695 and then continue and do the e-commerce business and all that. And it's a nightmare to go through the motion of getting just a certificate from them. And then you have to take that certificate and then put in your server. You have to take that certificate file and put that in the right location in your server and then restart your services. Then it will then project that padlock to the user. And that's a manual process. And you have to repeat the process every year. Ouch. No, I'm not doing that. You don't want to do that. And on top, you pay them money. No way. No. The simple solution is to use open technology and automate the end-to-end game of generating certificate whenever it is needed. And let us talk about the automation. But before that, the idea I want you to understand is that it's not only the money part that hurts, but the fact that I had to do this every year manually is the hard part. I don't want no manual methods. So that's the reason I hate it, honestly. I have been doing this manually for a couple of years, but then recently this thing came along and we are automated now. So what does automation involve? It means that you have to have a name, a DNS resolvable name. Once you have that name, you need to prove your ownership. Ownership proof has to be automated. Once that is proved, you are automatically issued a certificate through a script that you will see in a moment. And we'll run a service fairly quickly right now. So we'll just log out of this site and go to read this page where we have some documentation for it. And so here is getting started. So if you have shell access to a machine, you can basically run this sequence that they will describe in the documentation. What I have done is I think I have a one-liner written down already to make it happen. So that's what I'm trying to find. Hold on. I have written down a one-liner for us to play with in the exercise. It may be already on the website. Why did I forget that? It's probably there on the CloudGIS website in the architecture section already. So let's go see that. Yeah, here it is. So this segment already does what we just said it does. So let me send a link to you. And what this thing does is that you have to, first of all, have a machine with a name that results just like you know it does, which means you have to have some machine name on your domain mapped already. Like in this example, some other.cloudg.us is a name that I need to have a real machine running. How can I accomplish that? I can go to DigitalOcean, for example, and get a machine. And in there, I will start a droplet pretty much like this. Create one. Create Ubuntu. Five dollars is good. And San Francisco is nice. And select my key. And off we go with this name. Some other create. I created some other machine. It's getting created right now. I will soon get an IP address. When I get that IP address, I will go to my DNS server and make sure that it is assigned. And by the way, this is a one-time operation. So it is not that big a deal. But you might have this some other already, like I do. So some other. There is a DNS entry already for that. It currently points to this old IP address. I want to remove that and get this new IP address from here that I will copy and then take it and put it right there. And then save. So now my some other points to DigitalOcean. Bingo. So this machine is now a valid DNS name. Some other.cloudg.us. Let's go ping it. So here, this is a ping. Some other. Did I ping correctly? Yes, it does. Nice. Let us go connect. Did I connect? Yes, I'm in. Now what? So now that this connection is ready, what I need to be able to do is go back to our writeup and generate a standalone certificate. Pretty much like this. One-liner. So let us go see how does it work. So now we'll generate one. Let's encrypt. I think it needs to be installed. I think we need to be, this needs to be installed. So we don't have that installed. You need to first get that standalone certificate. The let's encrypt script has to be installed, which I don't have it on that box yet. So it will not run. However, we can install it by going to the site, bring that from that site called let's encrypt. Let's encrypt script. You can run that script manually by the getting started flow if you have. So you have to install this client called certbot client using whatever server you have. If you have none of those and you are on Linux operating system, you can get that script basically like this. So get, this is the way you install it. So you install that particular application called let's encrypt. And then you run that one-liner. And that's what I will do now from this line that I have written down here. Once I have run that application, let's encrypt ready. I will be able to play that one line after the installation finishes. It is installing the let's encrypt software on the machine right now. And it is finished, which means I can run this one line and get a certificate created like this. And it says, okay, I need to know who you are. And so here it is, that is me and says, please read the terms and conditions. And I said, fine. And so I say, I agree. And it says, okay, here, your certificate is ready now. That's it. Congratulations. Your certificate and chain has been saved at this location. You're done. Practically done. Let's go see what that certificate actually looks like. So we'll go to that location. So it is CDHC and then let's encrypt. And in there, we have a new bunch of folders. Let's go see the tree. Or rather, I go step by step to go to live folder. Sorry. Sorry. Sorry. Break it. Break it. Break it. CD live folder. And in there, you will have this file, this folder called some other. And I will open that folder some other. And in there, I will find a bunch of files, which are basically the certificate and the full chain with a private key stored for us, created for us. And this is a functional certificate valid, issued to us, automated through a certificate authority. Now these days known as let's encrypt. So we have it already now. How do we, how do we use it? How do we put this to use? So let's go read that in this site. We have constructed a certificate and placed it on our box in the right location. What we need to now do is if you want to construct a certificate, you know, it will expire at some time. So like, for example, the certificate says it, I think it said it will expire on some date in 2017. So you can read on the same command on a cron job. It will renew it for you. So just read on the claim command when you need to renew and that's it. It renews at that time, automatically looks at the existing certificate you might have as long as your DNS is correct and valid and shows ownership, it will succeed and automate this automating this via cron job will help you. Now, if you're using Docker, which is what I prefer to use, what I will do is construct a SSL certificate and place it inside the Docker container. And so what I will do now is basically run a bunch of Docker commands to construct an SSL certificate on the same box, like this box, pretty much like that, but this time through Docker. So here is one where I will install Docker. So I will install Docker on this box. And then I will run this Docker container to run a proxy server, nginx proxy, and then another container to run let's encrypt and then run our kutum example app that you probably know already. And so these three will actually create a new kutum example application that runs on the web browser inside your containers and you can access them through a web browser. So right now what we are doing is just quickly installing Docker on the same box and we will run a real application with a proper valid certificate in a matter of few minutes with three commands that generates three different containers. First one is this nginx proxy container. The second one is a let's encrypt companion container to generate certificates inside the Docker container. And then a third one that runs a real application in a third container. That's what we are looking to do in these three sequence step-by-step runs. So Docker just is getting installed right now on the same box. We will finish it up. Then I'll get ready with the proxy container right now. So here is the proxy container that we invoke, like this. So it will set up a proxy container, nginx proxy, that finishes up very quickly. Then we will have a let's encrypt container that will generate a certificate for our own box inside the container. So now we have a Docker container running already with the proxy. Next we will run a container to generate certificates. And that's what this container does. And so it will run in the background and now you have two containers running. One of them is the proxy and the other one is a certificate generation container, which is the let's encrypt container. And the third one is the actual application. This is a Tutum example application. And that's what I want to run in a third container. And that is running right now. It is actually getting installed and prepared. Once that thing runs correctly, what I need to do is open this website, some other.gini.us in a browser. So here we go and get ready to open it. As soon as this thing finishes, we should properly see a green padlock in that window in a browser. And it should show us a Tutum example application in a container right there in this window. As soon as the other thing finishes, we should see a padlock, an HTTPS by default. Off we go. And here it doesn't show you the padlock because we did not choose the HTTPS problem. You have to choose HTTPS and it says site cannot be reached. Refuse to connect. And there we have a padlock. And this padlock is showing you a certificate, which is, I think I should use a different browser. This Firefox will show you better. So I'm using a different browser and going to this some other site. And you see that the padlock shows. And we will see that it is verified by let's encrypt. We see more information. We have the certificate itself showing which says that this certificate has been issued to some other.gini.us. It begins on October 29th and it will expire on January 27th. So it's a three-month expiration. And every two months down the road, this other container, the let's encrypt container automatically goes and renews it for itself without a user having to do anything with the certificate generation process. It is end-to-end automated because of this let's encrypt companion container has the cron script running on the background. It runs every so often, every 60 days, looks at all the expiring certificates that are soon to expire and renews them by itself. If you don't renew, the service will actually give you an email notification that you know what, something went wrong. Your certificate did not renew. And so you will see that I will have an email coming from the service the last time, let's say some other. And so I may have deleted that email apparently or maybe not. Some other.cloud.gini.us. Let's see the certificate. I think I deleted that. But there was an email that came from let's encrypt that says, you did not renew the let's some other certificate. It expired. And that was true. I did not renew because I don't care. It's just a test. And that was this certificate I ran the last time. And it expired. And the container was not running. And that's the end-to-end automation example. This is how you should do it if you're doing it for any kind of usage, whether it is for a business, for a hobby, or just about anything automated. Don't spend money and use open tools. Not these companies. No, don't go there. Don't go there. Don't go anywhere else except that let's encrypt, which is open and automated. And I think with that, I want to take a moment to pause. We'll meet again Tuesday when it is next month. OK. Wow. November starts. So November 1st at 5 o'clock, we will meet again. And until that time, you enjoy let's encrypt if you like. And do the exercises that we have for you on that site. You will find them. And any questions are always welcome in the chat. We will cover VPN-related security the next time, which is going to be another hands-on exercise. And if you have other candidate ideas you want us to cover and discuss, please bring them on. Because I want to make sure that I actually address your specific needs. So I want to leave some time for us to discuss that. Specifically, if you have anything that you want us to cover, please bring that on. We will have content in our site that you want to explore yourself. I will cover VPN virtual private network connections between the cloud and your corporate data center the next time. So thank you, guys. Thank you very much. Thank you, sir. You're welcome, sir.