Sunday, April 16, 2017

Landing on The Moon using AI Bot

Recently I was playing around with OpenAI Gym and Keras Reinforcement Learning library (keras-rl). I was able to train an AI Agent for a task of landing on the Moon. 

OpenAI Gym provides all sorts of different environments to explore using AI bots. One of them is lunar lander, which I'll focus on here. 
Keras on the other hand is a high level library built on top of TensorFlow (or Theano). It provides mechanisms for constructing deep learning models easily. 

Creating a new environment using OpenAI Gym is as easy as this:

env = gym.make('LunarLander-v2')

Here's how you can add video recorder of progress during training:

env = gym.wrappers.Monitor(env, 
                           'recording', 
                           resume=True, 
                           video_callable=lambda count: count % record_video_every == 0
                           )

The model itself is quite simple DQN Agent with LinearAnnealedPolicy. The most important layer is Dense 512 neuron internal layer. It's responsible for understanding of the current situation during landing. Next small, dense layer on top of it is responsible for final decisions related to lunar lander actions (steering the engines).

Here's how it can be instantiated:

model = Sequential()
model.add(Flatten(input_shape=(1,) + env.observation_space.shape))
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dense(nb_actions))
model.add(Activation('linear'))
print(model.summary())

memory = SequentialMemory(limit=1000000, window_length=WINDOW_LENGTH)

policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1., value_min=.1, value_test=.05, nb_steps=1000000)

dqn = DQNAgent(model=model, nb_actions=nb_actions, policy=policy, memory=memory, nb_steps_warmup=50000, gamma=.99, target_model_update=10000, train_interval=4, delta_clip=1.)

dqn.compile(Adam(lr=.00025), metrics=['mae'])


Here's a summary of the model:

Layer (type)                     Output Shape          Param #     Connected to                 7    
flatten_1 (Flatten)              (None, 8)             0           flatten_input_1[0][0]            
dense_1 (Dense)                  (None, 512)           4608        flatten_1[0][0]                  
activation_1 (Activation)        (None, 512)           0           dense_1[0][0]                    
dense_2 (Dense)                  (None, 4)             2052        activation_1[0][0]               
activation_2 (Activation)        (None, 4)             0           dense_2[0][0]                    

Total params: 6,660
Trainable params: 6,660
Non-trainable params: 0


Training can take a lot of time. In this example I used 3.5 million steps.
The outcome gives quite reasonable behavior for lunar lander.
Note that depending on environment, learning can take less time (if environement is not very tricky). In this case I found that difficulties are related to sensitivity of the last phase of landing (touch down). It took some time for the model to figure this part out.

As a side note, OpenAI provides lunar lander Agent using optimal trajectory. In this example, the Agent polls environment to figure out actions that give highest output of Q function. 



37 comments:

  1. What you have written in this post is exactly what I have experience when I first started my blog.I’m happy that I came across with your site this article is on point,thanks again and have a great day.Keep update more information.
    Salesforce Training in Chennai
    SAS Training in Chennai
    Selenium Training in Chennai

    ReplyDelete
  2. Really Good blog post.provided a helpful information about landing on moon using AI bot.I hope that you will post more updates like this.
    MSBI Training in Chennai

    ReplyDelete
  3. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command
    best rpa training in bangalore
    rpa training in pune | rpa course in bangalore
    RPA training in bangalore
    rpa training in chennai

    ReplyDelete
  4. Really great post, Thank you for sharing This knowledge.Excellently written article, if only all bloggers offered the same level of content as you, the internet would be a much better place. Please keep it up!
    python Course in Pune
    python Course institute in Chennai
    python Training institute in Bangalore

    ReplyDelete
  5. Having an assignment due tomorrow and understand you can’t do it yourself? But you’re afraid of delegating your paper to some scam services as well? Why not check https://scamfighter.net/review/papersowl.com on Scamfighter.net

    ReplyDelete
  6. I am usually to blogging and i really admire your content. The article has really peaks my interest. I’m going to bookmark your web site and hold checking for brand spanking new information.
    speedy paper reviews

    ReplyDelete
  7. Its really helpful for the users of this site. I am also searching about these type of sites now a days. So your site really helps me for searching the new and great stuff.sap fiori Training in Bangalore

    ReplyDelete
  8. Very useful and information content has been shared out here, Thanks for sharing it.sap testing Training in Bangalore

    ReplyDelete
  9. I gathered a lot of information through this article.Every example is easy to undestandable and explaining the logic easily.sap gts Training in Bangalore

    ReplyDelete
  10. These provided information was really so nice,thanks for giving that post and the more skills to develop after refer that post.sap apo Training in Bangalore

    ReplyDelete
  11. Your articles really impressed for me,because of all information so nice.sap security Training in Bangalore

    ReplyDelete
  12. Linking is very useful thing.you have really helped lots of people who visit blog and provide them use full information.sap wm Training in Bangalore

    ReplyDelete
  13. Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving.sap srm Training in Bangalore

    ReplyDelete
  14. you are posting a good information for people and keep maintain and give more update too.
    Thanks and Regards : Python COurse in pune at 3ritechnologies

    ReplyDelete
  15. Nice post...Thanks for sharing...
    RPA Training in Bangalore - Marathahalli | RPA Training Institutes | RPA Course Fees and Content | RPA Interview Questions
    - ECare Technologies located in Marathahalli - Bangalore, is one of the best RPA Training institutes with 100% Placement support. RPA Training in Bangalore provided by RPA Certified Experts and real-time Working Professionals with handful years of experience in real time RPA Projects.

    ReplyDelete
  16. Blockchain Course in Bangalore with 100% placement. We are the Best Blockchain Course Institute in Bangalore. Our Agile testing course and Certification courses are taught by working professionals who are experts in Blockchain.

    Blockchain Training in Bangalore

    Blockchain course in bangalore

    Blockchain in bangalore

    Blockchain classes in bangalore

    Blockchain course institute in bangalore

    Agile testing course and Certification course syllabus

    best Blockchain course

    Blockchain course centers

    ReplyDelete
  17. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot.

    SAP Online Training

    SAP Classes Online

    SAP Training Online

    Online SAP Course

    SAP Course Online

    ReplyDelete
  18. I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you.

    SAP HANA Online Training

    SAP HANA Classes Online

    SAP HANA Training Online

    Online SAP HANA Course

    SAP HANA Course Online

    ReplyDelete
  19. Good Post! , it was so good to read and useful to improve my knowledge as an updated one, keep blogging. After seeing your article I want to say that also a well-written article with some very good information which is very useful for the AWS Cloud Practitioner Online Training

    ReplyDelete
  20. Just found your post by searching on the Google, I am Impressed and Learned Lot of new thing from your post.
    Sap Srm training in bangalore

    ReplyDelete
  21. This Post is really supportive to all of us. Eager that these kind of information you post in future also. Thanks for Sharing Otherwise if anyone want To Learn SAS Training
    :Italian Training in Bangalore

    ReplyDelete
  22. Virtual events are naturally more inclusive than in-person events in many respects but there are still a lot of things that planners can do to make them more accessible to people with disabilities. event marketing and Reasons Why AI Matchmaking is the Future for Events

    ReplyDelete
  23. I am sometimes inclined to believe as this concerns Keto Advanced Fat Burner. Keto Advanced Fat Burner is a gateway which leads in a new life. Just like there are different types of Keto Advanced Fat Burner, there are also different types of Keto Advanced Fat Burner. What can we do? We have to stay in the real world. There was no good reason for that. This is a considerable addition.


    Keto Advanced Fat Burner

    ReplyDelete
  24. In this article, I shall to provide you with an example of what I'm talking about in relation to Keto Advanced Fat Burner. This was foul smelling. Where can big wigs identify old Keto Advanced Fat Burner directions? I might want to discover detailed information in respect to Keto Advanced Fat Burner. I'm feeling lost right now. That isn't a case in point. This will be revisited soon.


    Keto Advanced
    Keto Advanced Weight Loss

    ReplyDelete
  25. Great info. Thank you for sharing this fascinating information together. I am so blessed to discover this. 카지노사이트

    ReplyDelete
  26. I like what you guys are up also. Such smart work and reporting! Carry on the superb works guys I have incorporated you guys to my blogroll. 카지노사이트

    ReplyDelete
  27. I found your blog while I was webs writing to find the above information. Your writing has helped me a lot. I'll write a nice post by quoting your post. 카지노사이트

    ReplyDelete
  28. Thanks for sharing the info. I located the information very useful. That’s a brilliant story you posted. I will come back to read some more. 카지노사이트

    ReplyDelete
  29. Thanks for this hi tech knowledge blog
    if anyone interested in SAP MM Training in Pune then join SAP course Now

    ReplyDelete
  30. Great info. Thank you for sharing this fascinating information together.
    Know about Software Testing

    ReplyDelete

  31. These websites are really needed, you can learn a lot 청마담

    ReplyDelete
  32. What a nice post! I'm so happy to read this. baccarat What you wrote was very helpful to me. Thank you. Actually, I run a site similar to you. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.


    ReplyDelete