python - Issue with scraping site with foreign characters -


i need scraper i'm writing. i'm trying scrape table of university rankings, , of schools european universities foreign characters in names (e.g. ä, ü). i'm scraping table on site foreign universities in exact same way, , works fine. reason, current scraper won't work foreign characters (and far parsing foreign characters, 2 scrapers same).

here's i'm doing try & make things work:

  1. declare encoding on first line of file:

    # -*- coding: utf-8 -*-

  2. importing & using smart unicode django framework django.utils.encoding import smart_unicode

    school_name = smart_unicode(html_elements[2].text_content(), encoding='utf-8',         strings_only=false, errors='strict').encode('utf-8')

  3. use encode function, seen above when chained smart_unicode function. can't think of else doing wrong. before dealing these scrapers, didn't understand different encoding, it's been bit of eye-opening experience. i've tried reading following, still can't overcome problem

i understand in encoding, every character assigned number, can expressed in hex, binary, etc. different encodings have different capacities how many languages support (e.g. ascii supports english, utf-8 supports seems. however, feel i'm doing necessary ensure characters printed correctly. don't know mistake is, , it's driving me crazy. please help!!

when extracting information web page, need determine character encoding, how browsers such things (analyzing http headers, parsing html find meta tags, , possibly guesswork based on actual data, e.g. presence of looks bom in encoding). can find library routine you.

in case, should not expect web sites utf-8 encoded. iso-8859-1 still in widespread use, , in general reading iso-8859-1 if utf-8 results in big mess (for non-ascii characters).


Comments

Post a Comment

Popular posts from this blog

java - Play! framework 2.0: How to display multiple image? -

i need display gallery of photos. here template: @(photos: list[photo]) @title = { <bold>gallery</bold> } @main(title,"photo"){ <ul class="thumbnails"> @for(photo <- photos) { <li class="span3"> <a href="#" class="thumbnail"> <img src="@photo.path" alt=""> </a> </li> } </ul> } and here controller method: public static result getphotos() { return ok(views.html.photo.gallery.render(photo.get())); } and here photo bean : @entity public class photo extends model { @id public long id; @required public string label; public string path; public photo(string path, string label) { this.path = path; this.label = label; } private static finder<long, photo> find = new finder<long, photo>( long.class, photo.class); public static list<phot...
Read more

gmail - Is there any documentation for read-only access to the Google Contacts API? -

i want read-only access user's google contacts. there stackoverflow member wanted this, , received following answer : yes, there is, use https://www.googleapis.com/auth/contacts.readonly scope , "view contacts". this exchange reference read-only access google contacts can find using google. simply using access token obtained attempt read usual google contacts api (i.e. https://www.google.com/m8/feeds ) fails insufficient permissions. similarly, using google contacts api on new target fails 404 errors (for example https://www.googleapis.com/auth/contacts.readonly/default/full ). is there documentation anywhere on how use api, or permissible way access google contacts api full read-write access?
Read more

php - Controller/JToolBar not working in Joomla 2.5 -

relevant code: jtoolbarhelper::custom('savecategories', 'save', '', 'save', false, false); ... <input type="hidden" name="controller" value="easyblogcontroller"> according can dig docs , own previous questions, should call savecategories() function in easyblogcontroller . i've tried setting value easyblog , easyblog.php (file name) current easyblogcontroller (class name). clicking on save button refreshes page. not redirect, echo or var_dump test code put in savecategories() function. var_dump ing jrequest::getvar('controller') , 'task' return correct values. creating controller object , using $controller->execute('task'); works. the values going depend on location of controller trying call, , have couple options here. typically there controller in base folder of component (likely components/com_easyblog) in file called controller.php , class name inside ...
Read more