Python3.6 Rename and Push Docker Images into New Repository

Below python code changes the repository names:


client = docker.from_env()
docker_api = docker.APIClient()
images = client.images.list()
for image in images:
    try:
        if image.tags[0] and '<image you want to rename>' in image.tags[0]: 
            version = image.tags[0].split("/")[-1]
            type(version)
            print("version is {}".format(version))
            docker_api.tag(image.tags[0],"<new repository>/{}".format(version))
    except Exception as Ex:
        print(image) 
After Renaming the images you can push these images in new repo:

docker images | grep <new repository> | awk '{print $1":"$2}' | xargs -L1 docker push 

Python Requests + Python Selenium in action

When using Scarpy or Requests in python for data scraping , sometimes we login to websites and then start scraping. Most of the time have less parameters to send in post request.

But Asp.Net website has too many parameters and its very annoying.  

The easy way to login a website is selenium.

Yes, when you login to a website , the website sets parameters on the browser in terms of session and cookies.

So by Getting login using selenium then extract it's cookies and set in python requests.

           

from selenium import webdriver
from lxml import html
import requests

def request(driver):  
                   s = requests.Session()
                   cookies = driver.get_cookies()
                   for cookie in cookies:
                         s.cookies.set(cookie['name'], cookie['value'])
                   return s
def login():
       driver = webdriver.Firefox()
       driver.get("http://www.example.com")
       driver.find_element_by_id('username').send_keys(username)
       driver.find_element_by_id('password').send_keys(password)
       driver.find_element_by_id('login').click()
     
       # Now move to other pages using requests
       req = request(driver)
       response = req.get("myurl")      
       print response.status_code
       htmlObj = html.fromstring(response.text)
       yourdata = htmlObj.xpath("your xpath")

login()

The above code is just for a concept. It will not run on your machine.

Login a website Using Scrapy

Scarpy is a great framework for scraping purpose. It has a very good structure for large projects and Oops style remove duplicate code.

This Post is just for education Purpose.

This code shows how to get login in a website using Scrapy:-

```def parse(self, response):
         yield FormRequest(posturl,
                                       formdata{'username':'yourUserName','Password':'!Z6Dj2hqaR'},
                                       callback=self.after_login)

    def after_login(self, response):
        yield Request("pageurl,callback=self.directory_page, meta={'page':1})
```
After this Scrapy takes care itself of session and cookies.







Installing mongodb on ubuntu 15

1.  sudo rm /etc/apt/sources.list.d/mongodb.list
2. echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
3.  sudo apt-get update
4.  sudo apt-get install -y mongodb-org

and then sudo service mongod start

Free Web Hosting for Web-Sites


Freehostingnoads.net


This web hosting site offers many great features to you free of charge. Some of the features include 20 GB space, 200 GB of traffic, 3 PHP versions, free site builder, POP3 email account, and even free domain hosting. The site also offers free tools that include script installer that has many different templates to take all the guess work out of making your own website. There will never be any ads or popups that will show up on your website ever.


ByetHost.com

This site also has many great features that help bring it to the top 5 list. The main features that this free webhosting site offers are 1000 MB disk space, FTP account, file manager, Control panel, free tech support, add on domain and sub-domains. When you register with Byethost.com you will be able to choose from many different domain names that can include your business name. This site also assures you that there will never be an ad or popup banner on your website.
Wix.com

This great web hosting site offers a lot of help when starting your own webpage. The best thing about this site if the easy to follow website builder, which will lead you step by step until you publish you website, which usually will only take a few minutes. You are able to choose a design from many different options and you can also watch a video that explains each step before you can begin. This site is very search engine friendly and will link all of you social networking sites right to it.
Yola.com

This web hosting site offers a free plan as well as 3 paid plans as well. The free plan is truly an amazing one that offers up to 3 pages of content, 1 GB bandwidth, 1 GB of storage, and a Yola subdomain name. It also offers a 30 minute or less web page builder and offers step by step instructions that are easy to follow.
X10hosting.com

This site has been around for a long time, since 2004 to be exact and offers many great features all for free. Some of the features this free site offers are community support, auto install scripts such as Joomla, WordPress and SMF, 3 email accounts per site. It also offers unlimited SQL, unmetered disk space and also unmetered data transfer. There are never any forced ads and there are no file size limits at all.
000WebHost.com

000webhost.com ($0.00 webhost), is an industry leader in providing top class free web hosting services without advertising! There are no hidden costs, no adverts, and no restrictive terms. Lightning fast speeds, maximum reliability and fanatical user support are just a few of the features you'll receive with our service

These free web hosting website do not provide freedom to the user. It result in a frustration. Most of the time your website is down or you can't use your domain name. 

I will Suggest you to use ONE.COM site for a shared hosting and for registering domain use CrazyDomains . The two sites have the best ration for price to money.

Comments are welcome!!!

Selenium Setup A-Z

Selenium Webdriver setup for Python is made available through Python Binding. This binding is facilitated by making use of available Python Installer and Easy Install Setup tool. Let us proceed with first automated Selenium Webdriver test using Python.
Step 1: Setup Python Environment
Python environment can be set in your system by downloading installation file available here (latest available version for 64 bit Windows as on writing this post is Python 2.7.8 Windows X86-64 MSI Installer)
Note: Download and install python installer as per your system requirement

Check the correctness of python environment setup by navigating to recently installed python directory and simply tying command ‘python‘ in command prompt.
Step 2: Setup Environment Path
Add the following path in system path:
1.  C:\Python27;
 2. C:\Python27\Scripts
Windows 8 users
1.       From the Desktop, right-click the very bottom left corner of the screen to get the Power User Task Menu.
2.       From the Power User Task Menu, click System.
3.       Click the Advanced System Settings link in the left column.
4.       In the System Properties window, click on the Advanced tab, then click the Environment Variables button near the bottom of that tab.
5.       In the Environment Variables window (pictured below), highlight the Path variable in the "System variables" section and click the Edit button. Add or modify the path lines with the paths you want the computer to access. Each different directory is separated with a semicolon as shown below.

C:\Python27;C:\Python27\Scripts



Note: You can edit other environment variables by highlighting the variable in the "System variables" section and clicking Edit. If you need to create a new environment variable, click New and enter the Variable name and Variable value.
Windows Vista and Windows 7 users
1.       From the Desktop, right-click the Computer icon and select Properties. If you don't have a Computer icon on your desktop, click the Start button, right-click the Computer option in the Start menu, and select Properties.
2.       Click the Advanced System Settings link in the left column.
3.       In the System Properties window, click on the Advanced tab, then click the Environment Variables button near the bottom of that tab.
4.       In the Environment Variables window (pictured below), highlight the Path variable in the "System variables" section and click the Edit button. Add or modify the path lines with the paths you want the computer to access. Each different directory is separated with a semicolon as shown below.

C:\Python27;C:\Python27\Scripts


Note: You can edit other environment variables by highlighting the variable in the "System variables" section and clicking Edit. If you need to create a new environment variable, click New and enter the Variable name and Variable value.
Windows 2000 and Windows XP users
The path is now managed by Windows 2000 and Windows XP and not the autoexec.bat or autoexec.nt files as was done with earlier versions of Windows. To change the system environment variables, follow the steps below.
1.       From the Desktop, right-click My Computer and click Properties. If you don't have a My Computer icon on your desktop, click the Start button, right-click the My Computer option in the Start menu, and select Properties.
2.       In the System Properties window, click on the Advanced tab.
3.       In the "Advanced" section, click the Environment Variables button.
4.       Finally, in the Environment Variables window (as shown below), highlight the Path variable in the Systems Variable section and click the Edit button. Add or modify the path lines with the paths you want the computer to access. Each different directory is separated with a semicolon as shown below.

C:\Program Files;C:\Winnt;C:\Winnt\System32


Note: You can edit other environment variables by highlighting the variable in the "System variables" section and clicking Edit. If you need to create a new environment variable, click New and enter the Variable name and Variable value.

Step 3: Install Python specific Selenium Package
Selenium bindings are readily available as package in Python. Python packages can be installed using any of methods described here.
We will use second and easiest method ‘Installing Globally’ of Selenium package installation. It requires setuptools installation and PATH definition in environmental variables of your system settings.
Setuptools as compressed version can be downloaded from here (latest available version as on writing this post is setuptools 0.9)
Uncompressing the setuptools downloaded and double click python file named ‘ez_setup’ available in extracted folder. This setup in turn will make easy_install.exe’ executable file automatically added to Scripts folder of python directory installed in step (1). Include exact path of Scripts directory in PATH of environmental variable.
Now navigate to Scripts directory of python installed and type the below command for Python Selenium package installation in command prompt.
‘easy_install selenium’

Allow command prompt to complete Python Selenium package installation.
Step 4: Installing Internet Explorer Server and Chrome Server
For running internet explorer and chrome you need to install the servers. User can download these from the following links:
IE :-
Chrome:-
Store it in Window executable path for example C:\Python27


Big Data and Python Programming Code


"""DATA SCOPE:There are 3 data files data_10.txt, data_11.txt, data_12.txt of class 10th, 11th, 12th respectively, which contains students data in
Given format (separated by pipe)

Id|Name|city|Grade|Age
s001|Rohan|mumbai|A|16

There is another file which contains scholarship Info (scholarship.txt) (not all the students get scholarship)

Id|Family_Income|No_Of_Members
s013|2000|10

find out all the scholarship getting students who got either C or D grade and who are from 'Mumbai' and whose Family_Income < 5000"""

import sys

def openFile(filename='',mode='r'):
   
    if (filename != ""):
        fileHandler = open(filename,mode)
        if fileHandler:
            return fileHandler
        else:
            print"\nFile was not opened\n"
            sys.exit()
    else:
        print"\n Please specify the filename"
        sys.exit()
       
def fileClose(fileHandler):
    fileHandler.close()
   
def processFile(fileHandler):
    perLineInformation = fileHandler.readlines()
   
    # It is a list contain all the lines in it.
    if(len(perLineInformation) > 0):
        return perLineInformation
    else:
        print"\n File is Empty"
       
def main(filename1,filename2='scholarship.txt'):
   
    file1 = openFile('filename1')   #opening student data file
    file2 = openFile('filename2')   #opening schorship.txt file
   
    file1List = processFile(file1)      #all lines are in a list of sentences
    file2List = processFile(file2)     
   
    listofID2 = []       
    for information2 in file2List:
        splitedInformation2 = information2.split('|')
       
        #As the id is common and student with scholarship less than 5000 is getting scholarship
       
        if ( splitedInformation2[1] < '5000'):                    #criteria1 is salary less than 5000 Rs.
            listofID2.append(splitedInformation2)                   #collected all id having salary less than 5000
   
    listofFilteredStudent =[]
   
    for information in file1List:
        splitedInformation = information.split('|')
       
        # criteria2 is grade must be 'C' or 'D' and Criteria3 is 'Must be resident of Mumbai' 
       
        if (splitedInformation[2]=='Mumbai' and (splitedInformation[3]=='C' or splitedInformation[3]=='D') and (splitedInformation[0] in listofID2)):
           
            listofFilteredStudent.append(information)
           
            outputFile = openFile('result.txt','a')
            outputFile.write(information)
            fileClose(outputFile)
           
    fileClose(file1)
    fileClose(file2)
###### Run this program by just entering the file name

main('data_10.txt')
main('data_11.txt')
main('data_12.txt')  

            

 
   
   

Youtube Comment Finder

life is short - you need Python!: Python Code to Search Youtube Comments: Today I am sharing a code that I wrote few months ago. It searches the comments in youtube video and stores those in a database (postgresql...