Python3.6 Rename and Push Docker Images into New Repository

Below python code changes the repository names:

client = docker.from_env()
docker_api = docker.APIClient()
images = client.images.list()
for image in images:
    try:
        if image.tags[0] and '<image you want to rename>' in image.tags[0]: 
            version = image.tags[0].split("/")[-1]
            type(version)
            print("version is {}".format(version))
            docker_api.tag(image.tags[0],"<new repository>/{}".format(version))
    except Exception as Ex:
        print(image) 

After Renaming the images you can push these images in new repo:

docker images | grep <new repository> | awk '{print $1":"$2}' | xargs -L1 docker push 

Python Requests + Python Selenium in action

When using Scarpy or Requests in python for data scraping , sometimes we login to websites and then start scraping. Most of the time have less parameters to send in post request.

But Asp.Net website has too many parameters and its very annoying.  

The easy way to login a website is selenium.

Yes, when you login to a website , the website sets parameters on the browser in terms of session and cookies.

So by Getting login using selenium then extract it's cookies and set in python requests.

           

from selenium import webdriver
from lxml import html
import requests

def request(driver):  
                   s = requests.Session()
                   cookies = driver.get_cookies()
                   for cookie in cookies:
                         s.cookies.set(cookie['name'], cookie['value'])
                   return s
def login():
       driver = webdriver.Firefox()
       driver.get("http://www.example.com")
       driver.find_element_by_id('username').send_keys(username)
       driver.find_element_by_id('password').send_keys(password)
       driver.find_element_by_id('login').click()
     
       # Now move to other pages using requests
       req = request(driver)
       response = req.get("myurl")      
       print response.status_code
       htmlObj = html.fromstring(response.text)
       yourdata = htmlObj.xpath("your xpath")

login()

The above code is just for a concept. It will not run on your machine.

Login a website Using Scrapy

Scarpy is a great framework for scraping purpose. It has a very good structure for large projects and Oops style remove duplicate code.

This Post is just for education Purpose.

This code shows how to get login in a website using Scrapy:-

```def parse(self, response):
         yield FormRequest(posturl,
                                       formdata{'username':'yourUserName','Password':'!Z6Dj2hqaR'},
                                       callback=self.after_login)

    def after_login(self, response):
        yield Request("pageurl,callback=self.directory_page, meta={'page':1})

```

After this Scrapy takes care itself of session and cookies.

Selenium Setup A-Z

Selenium Webdriver setup for Python is made available through Python Binding. This binding is facilitated by making use of available Python Installer and Easy Install Setup tool. Let us proceed with first automated Selenium Webdriver test using Python.

Step 1: Setup Python Environment

Python environment can be set in your system by downloading installation file available here (latest available version for 64 bit Windows as on writing this post is Python 2.7.8 Windows X86-64 MSI Installer)

Note: Download and install python installer as per your system requirement

Check the correctness of python environment setup by navigating to recently installed python directory and simply tying command ‘python‘ in command prompt.

Step 2: Setup Environment Path

Add the following path in system path:

1.  C:\Python27;

 2. C:\Python27\Scripts

Windows 8 users

1.       From the Desktop, right-click the very bottom left corner of the screen to get the Power User Task Menu.

2.       From the Power User Task Menu, click System.

3.       Click the Advanced System Settings link in the left column.

4.       In the System Properties window, click on the Advanced tab, then click the Environment Variables button near the bottom of that tab.

5.       In the Environment Variables window (pictured below), highlight the Path variable in the "System variables" section and click the Edit button. Add or modify the path lines with the paths you want the computer to access. Each different directory is separated with a semicolon as shown below.

C:\Python27;C:\Python27\Scripts

Note: You can edit other environment variables by highlighting the variable in the "System variables" section and clicking Edit. If you need to create a new environment variable, click New and enter the Variable name and Variable value.

Windows Vista and Windows 7 users

1.       From the Desktop, right-click the Computer icon and select Properties. If you don't have a Computer icon on your desktop, click the Start button, right-click the Computer option in the Start menu, and select Properties.

2.       Click the Advanced System Settings link in the left column.

3.       In the System Properties window, click on the Advanced tab, then click the Environment Variables button near the bottom of that tab.

4.       In the Environment Variables window (pictured below), highlight the Path variable in the "System variables" section and click the Edit button. Add or modify the path lines with the paths you want the computer to access. Each different directory is separated with a semicolon as shown below.

C:\Python27;C:\Python27\Scripts

Note: You can edit other environment variables by highlighting the variable in the "System variables" section and clicking Edit. If you need to create a new environment variable, click New and enter the Variable name and Variable value.

Windows 2000 and Windows XP users

The path is now managed by Windows 2000 and Windows XP and not the autoexec.bat or autoexec.nt files as was done with earlier versions of Windows. To change the system environment variables, follow the steps below.

1.       From the Desktop, right-click My Computer and click Properties. If you don't have a My Computer icon on your desktop, click the Start button, right-click the My Computer option in the Start menu, and select Properties.

2.       In the System Properties window, click on the Advanced tab.

3.       In the "Advanced" section, click the Environment Variables button.

4.       Finally, in the Environment Variables window (as shown below), highlight the Path variable in the Systems Variable section and click the Edit button. Add or modify the path lines with the paths you want the computer to access. Each different directory is separated with a semicolon as shown below.

C:\Program Files;C:\Winnt;C:\Winnt\System32

Note: You can edit other environment variables by highlighting the variable in the "System variables" section and clicking Edit. If you need to create a new environment variable, click New and enter the Variable name and Variable value.

Step 3: Install Python specific Selenium Package

Selenium bindings are readily available as package in Python. Python packages can be installed using any of methods described here.

We will use second and easiest method ‘Installing Globally’ of Selenium package installation. It requires setuptools installation and PATH definition in environmental variables of your system settings.

Setuptools as compressed version can be downloaded from here (latest available version as on writing this post is setuptools 0.9)

Uncompressing the setuptools downloaded and double click python file named ‘ez_setup’ available in extracted folder. This setup in turn will make ‘easy_install.exe’ executable file automatically added to Scripts folder of python directory installed in step (1). Include exact path of Scripts directory in PATH of environmental variable.

Now navigate to Scripts directory of python installed and type the below command for Python Selenium package installation in command prompt.

‘easy_install selenium’

Allow command prompt to complete Python Selenium package installation.

Step 4: Installing Internet Explorer Server and Chrome Server

For running internet explorer and chrome you need to install the servers. User can download these from the following links:

IE :-

 http://selenium-release.storage.googleapis.com/2.44/IEDriverServer_Win32_2.44.0.zip (32-bit)

http://selenium-release.storage.googleapis.com/2.44/IEDriverServer_x64_2.44.0.zip (64-bit)

Chrome:-

http://chromedriver.storage.googleapis.com/index.html?path=2.11/

Store it in Window executable path for example C:\Python27

Big Data and Python Programming Code

"""DATA SCOPE:There are 3 data files data_10.txt, data_11.txt, data_12.txt of class 10th, 11th, 12th respectively, which contains students data in
Given format (separated by pipe)

Id|Name|city|Grade|Age
s001|Rohan|mumbai|A|16

There is another file which contains scholarship Info (scholarship.txt) (not all the students get scholarship)

Id|Family_Income|No_Of_Members
s013|2000|10

find out all the scholarship getting students who got either C or D grade and who are from 'Mumbai' and whose Family_Income < 5000"""

import sys

def openFile(filename='',mode='r'):
   
    if (filename != ""):
        fileHandler = open(filename,mode)
        if fileHandler:
            return fileHandler
        else:
            print"\nFile was not opened\n"
            sys.exit()
    else:
        print"\n Please specify the filename"
        sys.exit()
       
def fileClose(fileHandler):
    fileHandler.close()
   
def processFile(fileHandler):
    perLineInformation = fileHandler.readlines()
   
    # It is a list contain all the lines in it.
    if(len(perLineInformation) > 0):
        return perLineInformation
    else:
        print"\n File is Empty"
       
def main(filename1,filename2='scholarship.txt'):
   
    file1 = openFile('filename1')   #opening student data file
    file2 = openFile('filename2')   #opening schorship.txt file
   
    file1List = processFile(file1)      #all lines are in a list of sentences
    file2List = processFile(file2)     
   
    listofID2 = []       
    for information2 in file2List:
        splitedInformation2 = information2.split('|')
       
        #As the id is common and student with scholarship less than 5000 is getting scholarship
       
        if ( splitedInformation2[1] < '5000'):                    #criteria1 is salary less than 5000 Rs.
            listofID2.append(splitedInformation2)                   #collected all id having salary less than 5000
   
    listofFilteredStudent =[]
   
    for information in file1List:
        splitedInformation = information.split('|')
       
        # criteria2 is grade must be 'C' or 'D' and Criteria3 is 'Must be resident of Mumbai' 
       
        if (splitedInformation[2]=='Mumbai' and (splitedInformation[3]=='C' or splitedInformation[3]=='D') and (splitedInformation[0] in listofID2)):
           
            listofFilteredStudent.append(information)
           
            outputFile = openFile('result.txt','a')
            outputFile.write(information)
            fileClose(outputFile)
           
    fileClose(file1)
    fileClose(file2)
###### Run this program by just entering the file name

main('data_10.txt')
main('data_11.txt')
main('data_12.txt')  

            

 
   
   

How to increase your net speed by removing ads on the browser page!!

Rs. 85 Cheap Web Hosting
Unlimited Data, Storage and Emails!
Unbelievable price for hosting. Go it.

Google Python Class Day 1 Part 1

Python Insider: Python 3.3.3 relased

Python Insider: Python 3.3.3 relased: A new maintenance release, Python 3.3.3 , has been released on November 13 2013.

Build Your Own Website
4000+ templates, customize and go live!

Recursive Scraping using different styles with MySQL Database.

Hi,
I don't have too much time to write Scrapy Tutorial for you people but this example may be a help full hand to the scrapy beginners.

this program recursively scrap youtube.com and the link for scraping comes from s database and the results store in database.

here is
Items.py
---------------------------------------------------------------------------------------------------------------------------------
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

from scrapy.item import Item, Field

class Project2Item(Item):
    # define the fields for your item here like:
    # name = Field()
    videoLink = Field()
    videoDuration= Field()
    imageVideo = Field()
    titleVideo = Field()
    Description = Field()
    view = Field()
    pass
---------------------------------------------------------------------------------------------------------------------

here is your pipelines.py
-------------------------------------------------------------------------------------------------------------------------------
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import MySQLdb
import re
class Project2Pipeline(object):
def __init__(self):
self.conn = MySQLdb.connect(host='localhost',
user ='root',
passwd = 'froot',
db = 'prime',
charset='utf8')
self.cursor = self.conn.cursor()
def process_item(self, item, spider):
try:
self.cursor.execute("""insert into scrapyTest (videoLink, videoTitle,videoView,flag)
VALUES (%s, %s, %s,1)""",(item['videoLink'],item['titleVideo'][0],item['view'][0]))
self.conn.commit()
except MySQLdb.Error, e:
        print "Error %d: %s" % (e.args[0], e.args[1])
 
         return item
--------------------------------------------------------------------------------------------------------------------------------
here is your settings.py
======================================================================
# Scrapy settings for project2 project
#
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#

BOT_NAME = 'project2'

SPIDER_MODULES = ['project2.spiders']
NEWSPIDER_MODULE = 'project2.spiders'
CONCURRENT_REQUESTS_PER_DOMAIN= 1
ITEM_PIPELINES = {
    'project2.pipelines.Project2Pipeline': 300,

}


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'project2 (+http://www.yourdomain.com)'
-------------------------------------------------------------------------------------------------------------------------------
here is your different spiders.py files:
========================================================================
from scrapy.spider import Spider
from scrapy.selector import Selector
from project2.items import Project2Item
import urlparse
from scrapy.http.request import Request
import MySQLdb
class spider(Spider):
name="Youtube"
allowed_domains = ["youtube.com"]
conn = MySQLdb.connect(host='localhost',user='root',passwd = 'froot',db = 'prime')
cursor = conn.cursor()
cursor.execute("select pageLink from scrapyTest where flag=2")
row = [item[0] for item in cursor.fetchall()]
print row
start_urls = row #["https://www.youtube.com/results?search_query=live+music+INdia&page=1"]
def parse(self,response):
sel = Selector(response)
next_page = sel.xpath("//div[@role='navigation']/a[@data-link-type='next']/@href").extract()
        listing = sel.xpath("//ol[@id='search-results']/li")
items=[]
for listi in listing:
item = Project2Item()
item['videoLink'] = "www.youtube.com"+listi.xpath("div[1]/a/@href").extract()[0]
item['videoDuration']=listi.xpath("div[1]/a/span[@class='video-time']/text()").extract()
item['imageVideo']=listi.xpath('div[1]/a/button//span/img/@src').extract()
item['titleVideo']=listi.xpath('div[2]/h3/a/@title').extract()
item['Description']=listi.xpath("div[2]/div[@class='yt-lockup-meta']/ul/li[3]/text()").extract()
item['view']= listi.xpath("div[2]/div[@class='yt-lockup-meta']/ul/li[3]/text()").extract()
items.append(item)
for item in items:
yield item
if not not next_page:
yield Request("http://www.youtube.com"+next_page[0],self.parse)

---------------------------------------------------------------------------------------------------------------------------------
second method:
========================================================================
from scrapy.selector import Selector
from scrapy.spider import Spider
from project2.items import Project2Item
from scrapy.http.request import Request

class youtubespider(Spider):
name = "youtube2"
allowed_domains = "www.youtube.com"
links=[]
for page in range(50):
x = 'https://www.youtube.com/results?search_query=live+music+concert+india&page=%s'%page
links.append(x)
start_urls = links
def parse(self,response):
sel = Selector(response)
sites = sel.xpath("//ol[@id='search-results']/li")
items=[]
for listi in sites:
item = Project2Item()
item['videoLink'] = "www.youtube.com"+listi.xpath("div[1]/a/@href").extract()[0]
item['videoDuration']=listi.xpath("div[1]/a/span[@class='video-time']/text()").extract()
item['imageVideo']=listi.xpath('div[1]/a/button//span/img/@src').extract()
item['titleVideo']=listi.xpath('div[2]/h3/a/@title').extract()
item['Description']=listi.xpath("div[2]/div[@class='yt-lockup-meta']/ul/li[3]/text()").extract()
item['view']= listi.xpath("div[2]/div[@class='yt-lockup-meta']/ul/li[3]/text()").extract()
items.append(item)
return items

=======================================================================
third method:
----------------------------------------------------------------------------------------------------------------------------------
from scrapy.selector import Selector
from project2.items import Project2Item
from scrapy.http.request import Request
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

class youtubespider(CrawlSpider):
name = "bollywoodMovies"
allowed_domains = ["www.youtube.com"]
start_urls = ['https://www.youtube.com/results?search_query=bollywood+movies&page=1']
rules = (Rule (SgmlLinkExtractor(allow=("page=\d+", )), callback="parseitem",follow=True,), )

def parseitem(self,response):
sel = Selector(response)
sites = sel.xpath("//ol[@id='search-results']/li")
items=[]
for listi in sites:
item = Project2Item()
item['videoDuration']=listi.xpath("div[1]/a/span[@class='video-time']/text()").extract()
item['videoLink'] = "www.youtube.com"+listi.xpath("div[1]/a/@href").extract()[0]
item['imageVideo']=listi.xpath('div[1]/a/button//span/img/@src').extract()
item['titleVideo']=listi.xpath('div[2]/h3/a/@title').extract()
item['Description']=listi.xpath("div[2]/div[2]/span/@data-original-html").extract()
item['view']= listi.xpath("div[2]/div[@class='yt-lockup-meta']/ul/li[3]/text()").extract()
items.append(item)
return items


========================================================================
fourth style: Try Youself

I hope this may be help full for you
----------------------------------------------------------------------------------------------------------------------------------

Selenium Example for iphone and android

from selenium import webdriver



# iPhone

driver = webdriver.Remote(browser_name="iphone", command_executor='http://172.24.101.36:3001/hub')



# Android

driver = webdriver.Remote(browser_name="android", command_executor='http://127.0.0.1:8080/hub')



# Google Chrome 

driver = webdriver.Chrome()



# Firefox 

driver = webdriver.Firefox()



# ------------------------------

# The actual test scenario: Test the codepad.org code execution service.



# Go to codepad.org

driver.get('http://codepad.org')



# Select the Python language option

python_link = driver.find_elements_by_xpath("//input[@name='lang' and @value='Python']")[0]

python_link.click()



# Enter some text!

text_area = driver.find_element_by_id('textarea')

text_area.send_keys("print 'Hello,' + ' World!'")



# Submit the form!

submit_button = driver.find_element_by_name('submit')

submit_button.click()



# Make this an actual test. Isn't Python beautiful?

assert "Hello, World!" in driver.get_page_source()



# Close the browser!

driver.quit()

Web Automation Using Python (Scrapy and Selenium Webdriver): Selenium 2 Link: Learn Selenium Using PythonScrap...

Web Automation Using Python (Scrapy and Selenium Webdriver):  Selenium 2 Link: Learn Selenium Using PythonScrap...:  Selenium 2 Link:  Learn Selenium Using Python Scrapy Link: Learn Scrapy Free Python E-Book:   DOWNLOAD ‎‎

 Selenium 2 Link: Learn Selenium Using Python

Scrapy Link: Learn Scrapy

Free Python E-Book:  DOWNLOAD‎‎

7. Using Proxy in Selenium Example

Hi  friends this is a perfect example of scrapy with proxy. Here  are four proxy which We are using alternatively.

from selenium import webdriver
import random
import time
from selenium.webdriver.common.proxy import *

while True:
    proxy = ['123.129.240.173:8081','201.63.184.41:3128','58.1.251.57:80','121.192.190.101:9999']
    myProxy = "http://"+proxy[random.randint(0,3)]
    print myProxy

    proxy = Proxy({
    'proxyType': 'MANUAL',
    'httpProxy': myProxy
  
    })

    driver = webdriver.Firefox(proxy=proxy)
    driver.implicitly_wait(30)
    driver.get("http://autopython.blogspot.in")
    time.sleep(5)
    driver.close()