Pythonic Discoveries, Part 2

While working on the feed parsing program, I wanted to display a numbered list. In C++, I would have used an incrementing variable and thought nothing of it. But I found this a little more complicated in Python. When I tried to simply set a variable and then display it along with an item in the list I was iterating through, I got an error that an int and a string cannot be concatenated. So I ended up doing it thus:

entrylist = []
for entry in current.entries:
 entrylist.append(entry.title)
bullet = 1
for x in entrylist:
  print str(bullet) + "- " + x
  bullet+=1

I don't know if this was a bad hack, and if there is an easier way to do this, but it worked. It seem interesting that an increment operator is not built into Python.\

Feed Parsing, Attempt 1

I wrote a basic program tonight which accepts an RSS feed, displays its entries, and gives the user the option to view a given entry's description. It's not very complex, but it allowed me to figure out some basic Python stuff, and I am pleased with how well it works.
Things to add:

• Escape out html in the description or display it differently.
  
• Make it work on immortalcuriosity.com instead of just the command line.
  
• Store the feeds to a database and give the option to refresh data for a feed previously entered.
  

I think once I get these 3 completed, I will have a good start on the first component of my system.

Oh, here's the code:

# A test program to learn about feedparser. It accepts a
# feed, displays its entries, and gives the option to
# display a given entries description. At least works with
# Slashdot and KurzweilAI feeds.

import feedparser

# Get the feed to parse
uri = raw_input("Please enter the feed to be parsed: ")

# Grab the feed
current = feedparser.parse(uri)

# Parse the feed
title = current.feed.title
description = current.feed.description

# Print data on the feed
print
print
print uri + " aka " + title + " is described by its owner as: "
print description + "."
print

# Store entry titles and print them
print "The current items are: "
entrylist = []
for entry in current.entries:
entrylist.append(entry.title)
bullet = 1
for x in entrylist:
print str(bullet) + "- " + x
bullet+=1
print

# Store item descriptions
entrydescs=[]
for desc in current.entries:
entrydescs.append(desc.description)

# See if any additional data is desired
contin = raw_input("Would you like to view any of those (Y or N)?: ")

# Find the item and print its description
if contin == "Y":
checkme = raw_input("Ok, which item do you want to view? ")
print
print entrylist[int(checkme)-1] + ": "
print entrydescs[int(checkme)-1]

Helpful Components

I discovered 2 helpful Python modules that will most likely prove very useful in my parsing project.

• Feedparser - This is a Python module which can parse a wide variety of the most common syndication formats. It is well documented, and seems well suited for the component I will need to take a feed and parse it according to fixed components.
• pyparsing - This module allows for the creation of grammars directly in Python code.

I also came across a powerful new technique for extracting information from text: text-mining. Instead of the tedious formation of grammars and topics through supervised learning, this technique uses "topic modeling" to form topics and appropriate divisions based on a system of combinations of words which are common.

Picture!

Although there may be different components added later, here is a basic diagram of the major components of the feed parsing system I aim to create.

A consideration of a personal business interaction

I wished for something today that at first surprised me, but made more sense in afterthought. At a grocery store, when it was my turn to checkout, I knew exactly what the cashier would be likely to say, the exact procedure she would follow. After the perfunctory greeting, she would ask whether I had a discount card for the particular store, then scan some or all of the items (depending on their number), then ask which bag type I would like. After all items were scanned and the price totalled, I would brandish my check card, which would prompt the query "Credit or debit?". Of course, I would select debit because I would not want to take the time to get the receipt and sign it when I could simply put in my PIN. In any case, since I selected debit, she would ask whether I could like any cash back, to which I would respond in the negative. After the transaction was completed, she would provide me with a receipt and I would be on my way. Nothing too extraordinary.

But this is what I will do every time, aside from the extremely rare cases of asking for an alternative bag type or for cash back, as a result of special circumstances. In today's case, as I walked out, I found myself thinking, "I sure wish I could set my grocery store config file to those responses as default, and pass special options when necessary." I most likely thought this as a result of being in a programming environment commonly, where seeking such improvements would be a matter of course. I smiled at my thought, but then asked "Wait, why not?" Why need I take my thought away from other considerations to respond whether I would like paper or plastic? Why would that be necessary more than once? Why not have my preferences for such things stored in my debit card, which I would slide through a reader after unpacking my items from the cart?

I think pondering such possibilties is fruitful, in that there is no reason that even small elements of everyday life such as this should not be improved when possible.

More Efficient Transportation

I considered recently, while sitting in traffic backed-up from a toll bridge, how a system could be implemented to solve such bottlenecks. It would be much preferable to have a national system of automatic tolls. Instead of having people sitting in booths taking cash, readers could be placed in such lanes, identical or similar to those in place for the "fast lanes" at tolls. Stickers with barcodes (or the equivalent patterns) could be distributed to every car owner via a national distribution, allowing for uniformaity and a better selection of unique identifiers.

Local governments could still be given the option to opt-in or not, setting their own timetable for integration, but the data would be prepared. The readers may be expensive, as well as the maintenance fee that should be charged to allow for participation in the program, but it seems the cost would definitely be less than maintaining a human workforce at the tolls.

Such a system would be far faster from the beginning, reduce traffic volume at such common chokepoints greatly, thus decreasing the likelihood for accidents, not to mention reducing the frustration of drivers greatly.

Pythonic discoveries, Part 1

The "item1.function(variable)" form in Python was confusing me, until I got something to work in the interpreter. I made a list:

>>> countries = ['USA', 'Russia', 'Cuba', 'Iceland', 'Greenland', 'Atlantis'])

and wanted to add 'France' to it. Trying "append('France')" did not work, as the append() function did not know where to act. However, "countries.append('France')" did work. I realized I had been taking the "x.y" notation as something more complex than it actually is. It can more easily understood after thinking about List Comprehension. List Comprehension takes an expression and applies a for conditional within it, followed by zero or more for and if conditionals. Thus in:

>>> num = [2, 4, 6]
>>> [3*x for x in num]
[6, 12, 18]
>>> [3*x for x in num if x > 3]
[12, 18]

the expression "3*x" is applied to each term x in the list "num".

When something is imported, say a module called "fruit", then one can say "import fruit", and "fruit.peel()" (assuming that peel was defined in "fruit"), the same x.y() form. The "x.y" means: "do, or look for, y in x, or in the context of x."

Adumbrations

I used to spend several hours a day combing through feeds, newsletters, and portals concerning topics I found interesting. At first, I merely noted them in memory. Then, as I became interested in being able to refer back to particular developments, I started a simple text file, divided into basic topics, such as "Security", "Programming Languages", etc. After a short time, this became unwieldy and ineffectual.

My next approach was to create an extensible folder structure, finely-grained by topic, containing document files which included the links to stories, tutorials, and other sources of information pertinent to the topic of their directory. I also found it easier to add personal notes and commentary on the information in this way. This method was much more effective, but still very limited.

As time progressed and my collection of information grew, I realized that the structure I had created was limited in two critical respects: 1) adding or editing information was overly time-consuming, as I had to find the appropriate document in the appropriate folder structure, then open, edit, and save it, 2) searching the structure was limited to the search functionality built into Windows, and later to Google Desktop search. It became clear at that stage that the design of my record system needed to be centered around 3 basic principles:

1. Ease of entering new information
2. Ease of searching for information
3. Ease of altering topical structure
   

Around the time when I began to grow dissatisfied with the folder/document system, I discovered Treepad. This is a wonderful program, with many potential uses. It allows one to form a structure of "trees" and "leaves" within a single file. Each of the leaves allow for insertion of text and images, and the entire structure can be altered, linked, and seached with ease.

Using Treepad was much more effective, and I would have been satisfied with it, had I not found Freemind. This allowed for all the features of Treepad, except that I was not limited to an Explorer-style visual interface, but rather a topic map which could be unfolded as desired. I found it much easier to find information, track developments, and understand topics through this display method than any structure I had used previously.

I ran into one final difficulty: as my topics of interest and my databank grew, I found that the time required to search through various feeds and portals, pick out interesting information, and record them in my topic structure was simply too great. Thus emerged the need for a system which I now set out to create. This system (for which I lack an effective name currently) will perform the following basic functions:

• Record an initial list of syndicated feeds and URLs of portals of interest, and allow for new feeds and sites to be imported
• Read each of the feeds or portals as appropriate and parse the information they contain
• Tag each piece of content based on an extensible topical structure
• Record content and applicable metadata in a database
• Provide an interface for the content to be searched based on a variety of criteria
  

The system may also be extended to include the following much more advanced features:

• Web spiders to find new feeds and sites which should be of interest based on the current topical structure and areas for which information is lacking and determined to be desired
• Visual interface for graphical display of topical structure and content relations

I hope to write this system mainly in Python, as it seems suit the task well in its simplicity and ability. This blog will serve as a record of the steps in its development, from things I learn about Python, to semantics, and probably much more.

Telos and Human Evolution

That natural biological evolution is progressive and goal-oriented, with man as its pinnacle, has been a common and influential notion. From ancient and learned sources such as Plotinus, to the widely recognized and copied illustration "March of Progress" by Rudolph Zallinger, the idea can be seen in scholarly literature and also in more quotidian descriptions of evolution and human development. Even up to the time of Darwin and up to the Modern Synthesis, the idea of there being a telos, or goal, of evolution was not uncommon. As biological explanations became more distant from theological sources, however, the idea of telos and of goals existing in the activity of nature began to wane.

In the studies of modern evolutionary biology, the notion of man as the natural apex has re-emerged in a new form. While there is still a connection made by some between goals and evolution in a religious context or, more commonly, in the context of religious individuals attempting to refute the claims of Darwinian evolution in specific, most of the modern usage of teleology has been in a descriptive way. E.g. it is not uncommon to hear an expression of the function of a particular organ with respect to the goal which it accomplishes.

But this descriptive shorthand is very different from the tendency that has sometimes been present in biology to characterize man, and the particular animal stock from which he comes, as superior to other forms of life, or as the goal of the process of evolution, or even as the goal of the entire universe. Some scientists reject such claims out of hand, calling it mere "mammal-centricity" or humanistic hubris (depending on the intended object of the claim) and insisting that the statistical nature of (Darwinian) evolution necessitates a development based only on various factors affecting survivability and not a development that can look to any external goal.

As for humans as they currently exist, some scientists claim that human evolution has come to a standstill. The biological reasoning for this is that for populations to develop new characteristics, the "populations have to be small and effectively isolated genetically from their relatives". This currently is not, and will not be for the foreseeable future, the case for humans in the modern world. Populations that were almost completely isolated centuries or even decades ago can now come into contact with members of populations all over the world. Since human groups do not now breed within completely isolated groups and do not stay in the same environment for geologically meaningful periods, it is very unlikely that they will develop characteristics sufficiently novel to ever form a new species, pending severe ecological disaster or displacement of portions of the population (such as in permanent space colonies).

However, there may be other senses in which man can develop. As described in Humanity and Willed Evolution, man alone can participate in his own development and guide it through the externalization of the primary means of survival. What allowed homo sapiens sapiens to advance to the place of dominance of earth was information, i.e. data properly processed and utilized. Man's rise was, admittedly, a combination of his nature and of natural circumstance. And some may claim that other species should be given the title of dominant over man. Bacteria, for instance, are much greater in number and live in a wider range of environments (in their natural state, i.e. without assistance). Man, however, is the only species that can externalize internal experience. The lessons learned by each member of the species can be shared with all of the other members, today far more than at any point in the past. There are cases of group learning occurring in other animal populations as well, most notably, perhaps, those of the japanese macaques. These ingenious creatures have learned how to sift wheat from sand, and have taken on more amusing characteristics such as playing with stones, in the few decades since researchers began interacting with them. But the spread of such behaviors is limited to individuals in regular contact, and seems to stay at a simplistic level.

Man's ability to formulate internal states in a way meaningful to other members of the species in a permanent way is what sets him apart. The ways and means of spreading these externalized states, information, have evolved far more rapidly than man's biological form since their emergence, and they have given him a far greater advantage with respect to survival than any biological development. Since the time when speech and writing developed, there have been relatively few physiological changes to man. What has changed is his intelligence and the availability of information. And there are now more humans on planet earth than at all historical periods combined. Man can live in space, under the ocean, and anywhere on the planet that he chooses, given enough effort. It seems at times that, given proper circumstances, man can say "Our will be done", and not feel boastful, but merely self-assured, in thinking that he can accomplish any feat.

But aside from questions of ability, the control of man's evolution by himself, both physically through genetic engineering, and mentally and socially through information exchange, makes man a completion of the evolutionary process in a certain way. Evolution occurs because statistical effects on populations in a species result from environmental conditions and changes. Man's scientific advancements make physical changes unnecessary when environmental conditions change, not to mention the fact that, more and more, man can change those very environmental conditions himself as well as make physical changes himself by will and not chance. This is an idea more encompassing than the three stages of technological evolution of Radovan Richta. Information is what gives man an edge, and a new kind of edge. It takes little imagination to imagine mankind in the not-too-distant future being in a position such that little save for inner turmoil on a drastic level or a universal shift such as the Big Crunch would threaten his physical existence.

This, then, is how mankind is understood as the end, or final achievement, of evolution: he alone can move the aspect that must change to fit his environment, the aspect of adaptation, outside of himself, through the creation of information. The movement can come to an even more complete fulfillment once it becomes possible for individual members of the human species to fully remove themselves from a biological medium. When this is achieved, along with the developments that it entails, man will no longer have to be concerned with the vast majority of concerns which drove his biological evolution. This will allow for a new type of man to emerge: homo sapiens sapiens will become homo excelsior, man in complete control of his form and his future. He will be truly only limited by his imagination, which needs have no restraints.

Humanity and Willed Evolution

New methods of expression and dissemination of information that have emerged in the recent times that allow his nature as a force of improving the efficiency of his own development to more easily seen. Since intellection and expression of abstract ideas developed in man, this potential existed. From the articulations of philosophers and other thinkers in earlier periods of civilization, e.g. in Ancient Greece, it appears that, in many cases, humanity viewed the universe as something eternal. As soon as the idea of an origin arises, however, the idea that all things might have had a beginning, the idea becomes captivating. It is almost as if man had to develop the idea, and it took on its own internal force.

Once this notion is achieved, man will consider the idea of a variability of form following this origin, if only as a possibility. The idea of things having a beginning and of the state and quality of things being able to change moved from a mere logical possibility, however, to an assumed entity with its own power. Somehow, man arrived at the idea of a progression, that things can be changed for the better in some way. And just as statistical and biological forces cause changes over time in living systems that are beneficial with respect to preservation in some way, so can this happen for man in society, and even personally, in each man’s thoughts. And since man has a will toward his own survival, he can consider altering his own development. This may occur unconsciously, socially, or otherwise at first. But once 2 other factors are present, development can greatly increase:

1. man must decide that he can and will take his own development in hand,


2. man changes the material that development works on from a biological to an informational form.

One of the primary virtues of informational versus biological evolution is that selection in the former can occur independent of individuals. Death has a different meaning in such a context. During the current age, the rate of development in terms of information is much greater than that caused by biological change, mutations, etc. This causes a shift in the understanding of what forms and wholes are, what individuals are, what survival is, etc. Books, articles, networks, even conversations and the ideas that they foster become the new creatures that interact, and survive and reproduce or are destroyed. This change can occur much more readily than its biological analog, and, more importantly, it can be willed. Thus man forms a new medium for his development. It may even be possible for him to transfer all of his content, or at least all of the intellectual content he wishes to preserve, into a non-biological form. He may be able to achieve a manifestation completely based in information and not matter. In this way he could alter his form in any way he wished with extreme rapidity.