This is our first post and we are excited about this.
Clouderizer offers a platform to address various road blocks & challenges involved in developing deep learning models, like
- Cost effective Cloud infrastructure,
- Hassle free Environment setup,
- Secure and Easy Remote access
- Cloud based workspace for easy modifications and iterations
Clouderizer tries to bring together all these aspects of Deep Learning development under one umbrella as Clouderizer Projects.
On first screen of wizard, you can assign your project a name. You can also navigate to Community Projects from bottom of this screen to clone ready to use Project Templates from Clouderizer Community. There are some very useful Deep Learning templates available that can get you started in minutes.
After cloning, you are taken back to project wizard with all fields pre-populated. You can change the name of your project if you want. Press Next to proceed.
Setup page allows us to setup the development environment for our Deep Learning project. Clouderizer by default sets up Anaconda3, python 3.6 environment. We can specify list of pre-requisite Debian, Conda and PIP packages. Need for remote access to SSH and Jupyter Notebooks can also be specified here.
Next comes Workspace. Every Clouderizer project has 3 important folders.
- Data – Training data or any other kind of data needed for your code. You can specify to either upload this data from your local drive, download it from some Git repository or, in case data is already present in Clouderizer Drive, you can specify the relative path in your Clouderizer Drive. On start of project, this data is automatically downloaded in data directory of target machine.
- Code – Your project code defining your model and training instructions. Here also, you can specify to either upload the code from your local drive, or download it from some Git repositary, or specify a relative path in your Clouderizer Drive. On start of project, this is automatically downloaded in code directory of target machine. Once you start working on the machine, any local changes in Code are synced to Clouderizer Drive in real time. This allows you to easily resume your work later on same or different hardware.
- Output – This is where output of your Deep Learning models should go. Any intermediate checkpoints should also be saved here. Once project starts, Clouderizer will sync this folder, in real time, with Clouderizer Drive. This will make sure in event of some interruption, your training can resume easily from where it left.
You can also specify a startup script here which can auto-trigger your training process once the project starts.
Next comes Machine. You can choose between
- Local Machine – If you select this option, you will need shell access to the target machine to start Clouderizer project manually. This can be your local machine itself, or some machine (or VM) in your local network, or some cloud machine for which you have shell access.
- AWS EC2 Spot Instance – This option requires you to link your Clouderizer account with your AWS account. Detail steps for this linking can be found in this blog. Once linked, you can specify various specs for your target machine, like AMI, OS, storage size, machine type (hardware specifications). This also auto recommends the best bidding price for the required configuration across all AWS regions.
Thats it! Pressing Next shows you a summary of what you have configured allowing you to verify and save the project.
Now this project shows up in main project list. You can press Start against the project to get started.
For Local Machine projects, you will be presented with shell snippet to manually start the project. You can login to the target machine and paste above command on the shell prompt. This will download and install Clouderizer service on the machine and might take some time depending on number of pre-requisites specified in the project. Once command completes, you can login back to Clouderizer Console and see the project status as Running.
At this point, target machine must be ready with all pre-requisites and environment setup, data and code folder provisioned as per project setting, and startup scripts, if any, trigered.
You can access remote SSH Terminal and Jupyter Notebook from the project buttons. Jupyter Notebook will have code directory as its home.
You can view progress of your startup command (that you specified at time of creation of project) by typing the following command in remote SSH window.
tmux a -t startup
At any point of time, you can navigate to Clouderizer Drive from Main menu on left panel. You can go inside project folder and view code, data and out directory. Code and Out directory on Clouderizer Drive gets synced in real time with target machine while Clouderizer project is running.
Hopefully this article will help you get started with Clouderizer. In case you have any questions, feel free to discuss it our community. We will try our best to answer as soon as possible.