Personal Photography 2.0

I have been considering solutions to the data overload problem that I mentioned in my last post. For now, I will hold off on dealing with paper documents and the issue of multiple online content repositories. Concerning digital images, however, here is an example of what I would consider a highly optimized system which could be created using current technology.

The image capture device itself need not be any different than current digital cameras except in a single essential and several optional aspects. The essential aspect is wireless data transfer capability. As soon as a picture is taken (and perhaps pending a 5-20 second waiting period to allow for deletion of images locally that are not wanted), it should be transferred wirelessly and automatically through the internet to either a personal computer or, perhaps preferably, to a hosted server or service. The latter would likely be better since a hosted service would have a higher percentage of uptime (unless the individual has a very stable machine running with a UPS and their ISP never goes down). Now even at this point, before any of the "magical", i.e. extremely helpful and cool, elements come into play, a great benefit has been realized. Viz.:

• If anything happens to the camera or memory stick while out, the data is preserved. From dropping a camera to the memory becoming corrupted, things can happen.

The camera and the memory stick are no longer single points of failure. The data captured will be preserved almost immediately after capture such that it cannot easily be lost.

In addition to this, however, one can imagine many other potential features that could be added to the system. The camera could be fitted with additional sensors, or additional data could be associated with the images captured. Normally, digital cameras simply associate the time of the shot with the file created. But the temperature, say, or weather conditions, or altitude, could also be added as metadata. This would allow for a more coherent whole of the events captured to be constructed later. Of greatest assistance, I think, would be GPS-based location data for each image. This could be incorporated into the wireless data transfer system added to the camera. In this way, a real "trip" could be reconstructed from the images, via a mashup with Google Maps, for instance.

The final elements of this system would be performed by the user's computer or the hosted service, for performance reasons. The simplest step at this point would be to display the images captured in a web portal, associated and organized by their metadata, time, location, etc. In this fashion, one could make images from a vacation or outing available while still out on the trip! And that without having to download them to a laptop and have internet access on it at the time. Or if one wished the images to be kept private, one would minimally already have a photo album waiting upon returning from the trip.

The more complicated, but also more interesting, step now would be to analyze the images for content and provide them with appropriate names. This could utilize preferences set by the user concerning the naming, length, title versus tags, etc. Automated image recognition technology has not been deployed for individual use in many capacities as of yet, but this feature would provide for great strides in personal organization. Instead of sorting through hundreds of images and coming up with names for them such that they can be searched, imagine an automated process which would scan the images, find and assign likely tags or titles, and organize the images into albums divided by topic or by location!

I currently lack the skill to implement the steps of this process, but this entire system could be realized based on presently-existing commercial technology.

Data Overload

Month by month, I have increasing difficulty with an overload of data. Case in point: I just purchased a digital camera for myself and Anna. It will be great to have a photo-record of our life at this time, trips we take, etc. But, getting that record is one thing (a rather simple thing with the amazing features of current digital cameras). Organizing that record into a form that is at all useful is quite another. We just finished a trip to visit my grandparents over Christmas (along with touring Williamsburg, VA and nearby historic sites), and came back with around 400 images. All dated and of great quality yes, but with useless names, and completely unorganized.

I imported them all onto my computer quickly, no longer worrying about storage space, as I have a relatively immense amount and can obtain more easily. Having gotten even this far is much better than nothing. But, I still need to rename all of the images to useful titles and organize them in folders thematically (or do so with an organizational program), to be near a point where I can repeatedly view and use those images with ease.

This is similar to the problem I face with the piles of papers and notebooks that I have collected over the years and now wish to digitize. Some of these contain thoughts and ideas that either I believe to still hold value in themselves, or which would minimally be instructive in developing an understanding of how my thought progressed to its current state. But to be such, I would need to normalize and order them.

Additionally, the same conundrum arises on an everyday basis, with an annoying additional issue. I now record thoughts and observations in a variety of digital locations, from Google Docs to Yahoo! Notebook, etc. These too must be unified and ordered to be of use. Now, however, that I fully recognize this problem, my thought is hampered by approaching its solution. I spend time thinking about how to better organize and catalog my records, and this takes away from the time I would otherwise spend thinking of and creating those records.

More on possible solutions next.

Feedorific design, part 1: Feedreader

Here are my thoughts on the design of Feedorific, Django-version:

The various apps will be:

• Feedreader -
  Accepts and stores feeds from users, gets the xml, parses out and displays entries and descriptions. Feeds entered will also be stored to DB.
  
• Structureparser -
  Parses stored feeds for their structure.
  
• Contentparser -
  Parses stored feeds for their content.
  
• Organization -
  Very unsure about this last section's design. It will display the fully-parsed feeds, allow searching, tagging, etc. This may also be integrated with a visually-organized display of feed articles by content.

Here is what I have on the design of the feedreader element so far:

Feedreader design

Django to the rescue

I found myself having much difficulty getting what I had of the feedparser setup in such a way that I could return html code that looked good. Also, there are no real tutorials, since no one seems to just use Python and html. It seems a lot more common to use some templating language or some other framework to link everything.

So, I decided to rewrite it all with Django. I don't know Django very well, but it seems powerful enough without having to be another enterprise level CMS that I don't need. It is also well-documented, Python-centric, and designed for fast-paced development.

It's (a)live! Feedorific!

Since the old immortalcuriosity.com was all in Plone, after switching to a different server, I had a blank slate to work with. What better to fill this void with than my feedparsing project! So with a little Apache, a little Python, and a little help from my friends, the current form of the feedparsing system is accessible at www.immortalcuriosity.com. A warning: It looks like junk and barely works in IE. It's perfect in Firefox. Sorry, but I just don't care about a bad browser enough to put the hacks in yet. Maybe later.

Pythonic Discoveries, Part 2

While working on the feed parsing program, I wanted to display a numbered list. In C++, I would have used an incrementing variable and thought nothing of it. But I found this a little more complicated in Python. When I tried to simply set a variable and then display it along with an item in the list I was iterating through, I got an error that an int and a string cannot be concatenated. So I ended up doing it thus:

entrylist = []
for entry in current.entries:
 entrylist.append(entry.title)
bullet = 1
for x in entrylist:
  print str(bullet) + "- " + x
  bullet+=1

I don't know if this was a bad hack, and if there is an easier way to do this, but it worked. It seem interesting that an increment operator is not built into Python.\

Feed Parsing, Attempt 1

I wrote a basic program tonight which accepts an RSS feed, displays its entries, and gives the user the option to view a given entry's description. It's not very complex, but it allowed me to figure out some basic Python stuff, and I am pleased with how well it works.
Things to add:

• Escape out html in the description or display it differently.
  
• Make it work on immortalcuriosity.com instead of just the command line.
  
• Store the feeds to a database and give the option to refresh data for a feed previously entered.
  

I think once I get these 3 completed, I will have a good start on the first component of my system.

Oh, here's the code:

# A test program to learn about feedparser. It accepts a
# feed, displays its entries, and gives the option to
# display a given entries description. At least works with
# Slashdot and KurzweilAI feeds.

import feedparser

# Get the feed to parse
uri = raw_input("Please enter the feed to be parsed: ")

# Grab the feed
current = feedparser.parse(uri)

# Parse the feed
title = current.feed.title
description = current.feed.description

# Print data on the feed
print
print
print uri + " aka " + title + " is described by its owner as: "
print description + "."
print

# Store entry titles and print them
print "The current items are: "
entrylist = []
for entry in current.entries:
entrylist.append(entry.title)
bullet = 1
for x in entrylist:
print str(bullet) + "- " + x
bullet+=1
print

# Store item descriptions
entrydescs=[]
for desc in current.entries:
entrydescs.append(desc.description)

# See if any additional data is desired
contin = raw_input("Would you like to view any of those (Y or N)?: ")

# Find the item and print its description
if contin == "Y":
checkme = raw_input("Ok, which item do you want to view? ")
print
print entrylist[int(checkme)-1] + ": "
print entrydescs[int(checkme)-1]

Helpful Components

I discovered 2 helpful Python modules that will most likely prove very useful in my parsing project.

• Feedparser - This is a Python module which can parse a wide variety of the most common syndication formats. It is well documented, and seems well suited for the component I will need to take a feed and parse it according to fixed components.
• pyparsing - This module allows for the creation of grammars directly in Python code.

I also came across a powerful new technique for extracting information from text: text-mining. Instead of the tedious formation of grammars and topics through supervised learning, this technique uses "topic modeling" to form topics and appropriate divisions based on a system of combinations of words which are common.

Picture!

Although there may be different components added later, here is a basic diagram of the major components of the feed parsing system I aim to create.

A consideration of a personal business interaction

I wished for something today that at first surprised me, but made more sense in afterthought. At a grocery store, when it was my turn to checkout, I knew exactly what the cashier would be likely to say, the exact procedure she would follow. After the perfunctory greeting, she would ask whether I had a discount card for the particular store, then scan some or all of the items (depending on their number), then ask which bag type I would like. After all items were scanned and the price totalled, I would brandish my check card, which would prompt the query "Credit or debit?". Of course, I would select debit because I would not want to take the time to get the receipt and sign it when I could simply put in my PIN. In any case, since I selected debit, she would ask whether I could like any cash back, to which I would respond in the negative. After the transaction was completed, she would provide me with a receipt and I would be on my way. Nothing too extraordinary.

But this is what I will do every time, aside from the extremely rare cases of asking for an alternative bag type or for cash back, as a result of special circumstances. In today's case, as I walked out, I found myself thinking, "I sure wish I could set my grocery store config file to those responses as default, and pass special options when necessary." I most likely thought this as a result of being in a programming environment commonly, where seeking such improvements would be a matter of course. I smiled at my thought, but then asked "Wait, why not?" Why need I take my thought away from other considerations to respond whether I would like paper or plastic? Why would that be necessary more than once? Why not have my preferences for such things stored in my debit card, which I would slide through a reader after unpacking my items from the cart?

I think pondering such possibilties is fruitful, in that there is no reason that even small elements of everyday life such as this should not be improved when possible.

More Efficient Transportation

I considered recently, while sitting in traffic backed-up from a toll bridge, how a system could be implemented to solve such bottlenecks. It would be much preferable to have a national system of automatic tolls. Instead of having people sitting in booths taking cash, readers could be placed in such lanes, identical or similar to those in place for the "fast lanes" at tolls. Stickers with barcodes (or the equivalent patterns) could be distributed to every car owner via a national distribution, allowing for uniformaity and a better selection of unique identifiers.

Local governments could still be given the option to opt-in or not, setting their own timetable for integration, but the data would be prepared. The readers may be expensive, as well as the maintenance fee that should be charged to allow for participation in the program, but it seems the cost would definitely be less than maintaining a human workforce at the tolls.

Such a system would be far faster from the beginning, reduce traffic volume at such common chokepoints greatly, thus decreasing the likelihood for accidents, not to mention reducing the frustration of drivers greatly.

Pythonic discoveries, Part 1

The "item1.function(variable)" form in Python was confusing me, until I got something to work in the interpreter. I made a list:

>>> countries = ['USA', 'Russia', 'Cuba', 'Iceland', 'Greenland', 'Atlantis'])

and wanted to add 'France' to it. Trying "append('France')" did not work, as the append() function did not know where to act. However, "countries.append('France')" did work. I realized I had been taking the "x.y" notation as something more complex than it actually is. It can more easily understood after thinking about List Comprehension. List Comprehension takes an expression and applies a for conditional within it, followed by zero or more for and if conditionals. Thus in:

>>> num = [2, 4, 6]
>>> [3*x for x in num]
[6, 12, 18]
>>> [3*x for x in num if x > 3]
[12, 18]

the expression "3*x" is applied to each term x in the list "num".

When something is imported, say a module called "fruit", then one can say "import fruit", and "fruit.peel()" (assuming that peel was defined in "fruit"), the same x.y() form. The "x.y" means: "do, or look for, y in x, or in the context of x."

Adumbrations

I used to spend several hours a day combing through feeds, newsletters, and portals concerning topics I found interesting. At first, I merely noted them in memory. Then, as I became interested in being able to refer back to particular developments, I started a simple text file, divided into basic topics, such as "Security", "Programming Languages", etc. After a short time, this became unwieldy and ineffectual.

My next approach was to create an extensible folder structure, finely-grained by topic, containing document files which included the links to stories, tutorials, and other sources of information pertinent to the topic of their directory. I also found it easier to add personal notes and commentary on the information in this way. This method was much more effective, but still very limited.

As time progressed and my collection of information grew, I realized that the structure I had created was limited in two critical respects: 1) adding or editing information was overly time-consuming, as I had to find the appropriate document in the appropriate folder structure, then open, edit, and save it, 2) searching the structure was limited to the search functionality built into Windows, and later to Google Desktop search. It became clear at that stage that the design of my record system needed to be centered around 3 basic principles:

1. Ease of entering new information
2. Ease of searching for information
3. Ease of altering topical structure
   

Around the time when I began to grow dissatisfied with the folder/document system, I discovered Treepad. This is a wonderful program, with many potential uses. It allows one to form a structure of "trees" and "leaves" within a single file. Each of the leaves allow for insertion of text and images, and the entire structure can be altered, linked, and seached with ease.

Using Treepad was much more effective, and I would have been satisfied with it, had I not found Freemind. This allowed for all the features of Treepad, except that I was not limited to an Explorer-style visual interface, but rather a topic map which could be unfolded as desired. I found it much easier to find information, track developments, and understand topics through this display method than any structure I had used previously.

I ran into one final difficulty: as my topics of interest and my databank grew, I found that the time required to search through various feeds and portals, pick out interesting information, and record them in my topic structure was simply too great. Thus emerged the need for a system which I now set out to create. This system (for which I lack an effective name currently) will perform the following basic functions:

• Record an initial list of syndicated feeds and URLs of portals of interest, and allow for new feeds and sites to be imported
• Read each of the feeds or portals as appropriate and parse the information they contain
• Tag each piece of content based on an extensible topical structure
• Record content and applicable metadata in a database
• Provide an interface for the content to be searched based on a variety of criteria
  

The system may also be extended to include the following much more advanced features:

• Web spiders to find new feeds and sites which should be of interest based on the current topical structure and areas for which information is lacking and determined to be desired
• Visual interface for graphical display of topical structure and content relations

I hope to write this system mainly in Python, as it seems suit the task well in its simplicity and ability. This blog will serve as a record of the steps in its development, from things I learn about Python, to semantics, and probably much more.

Telos and Human Evolution

That natural biological evolution is progressive and goal-oriented, with man as its pinnacle, has been a common and influential notion. From ancient and learned sources such as Plotinus, to the widely recognized and copied illustration "March of Progress" by Rudolph Zallinger, the idea can be seen in scholarly literature and also in more quotidian descriptions of evolution and human development. Even up to the time of Darwin and up to the Modern Synthesis, the idea of there being a telos, or goal, of evolution was not uncommon. As biological explanations became more distant from theological sources, however, the idea of telos and of goals existing in the activity of nature began to wane.

In the studies of modern evolutionary biology, the notion of man as the natural apex has re-emerged in a new form. While there is still a connection made by some between goals and evolution in a religious context or, more commonly, in the context of religious individuals attempting to refute the claims of Darwinian evolution in specific, most of the modern usage of teleology has been in a descriptive way. E.g. it is not uncommon to hear an expression of the function of a particular organ with respect to the goal which it accomplishes.

But this descriptive shorthand is very different from the tendency that has sometimes been present in biology to characterize man, and the particular animal stock from which he comes, as superior to other forms of life, or as the goal of the process of evolution, or even as the goal of the entire universe. Some scientists reject such claims out of hand, calling it mere "mammal-centricity" or humanistic hubris (depending on the intended object of the claim) and insisting that the statistical nature of (Darwinian) evolution necessitates a development based only on various factors affecting survivability and not a development that can look to any external goal.

As for humans as they currently exist, some scientists claim that human evolution has come to a standstill. The biological reasoning for this is that for populations to develop new characteristics, the "populations have to be small and effectively isolated genetically from their relatives". This currently is not, and will not be for the foreseeable future, the case for humans in the modern world. Populations that were almost completely isolated centuries or even decades ago can now come into contact with members of populations all over the world. Since human groups do not now breed within completely isolated groups and do not stay in the same environment for geologically meaningful periods, it is very unlikely that they will develop characteristics sufficiently novel to ever form a new species, pending severe ecological disaster or displacement of portions of the population (such as in permanent space colonies).

However, there may be other senses in which man can develop. As described in Humanity and Willed Evolution, man alone can participate in his own development and guide it through the externalization of the primary means of survival. What allowed homo sapiens sapiens to advance to the place of dominance of earth was information, i.e. data properly processed and utilized. Man's rise was, admittedly, a combination of his nature and of natural circumstance. And some may claim that other species should be given the title of dominant over man. Bacteria, for instance, are much greater in number and live in a wider range of environments (in their natural state, i.e. without assistance). Man, however, is the only species that can externalize internal experience. The lessons learned by each member of the species can be shared with all of the other members, today far more than at any point in the past. There are cases of group learning occurring in other animal populations as well, most notably, perhaps, those of the japanese macaques. These ingenious creatures have learned how to sift wheat from sand, and have taken on more amusing characteristics such as playing with stones, in the few decades since researchers began interacting with them. But the spread of such behaviors is limited to individuals in regular contact, and seems to stay at a simplistic level.

Man's ability to formulate internal states in a way meaningful to other members of the species in a permanent way is what sets him apart. The ways and means of spreading these externalized states, information, have evolved far more rapidly than man's biological form since their emergence, and they have given him a far greater advantage with respect to survival than any biological development. Since the time when speech and writing developed, there have been relatively few physiological changes to man. What has changed is his intelligence and the availability of information. And there are now more humans on planet earth than at all historical periods combined. Man can live in space, under the ocean, and anywhere on the planet that he chooses, given enough effort. It seems at times that, given proper circumstances, man can say "Our will be done", and not feel boastful, but merely self-assured, in thinking that he can accomplish any feat.

But aside from questions of ability, the control of man's evolution by himself, both physically through genetic engineering, and mentally and socially through information exchange, makes man a completion of the evolutionary process in a certain way. Evolution occurs because statistical effects on populations in a species result from environmental conditions and changes. Man's scientific advancements make physical changes unnecessary when environmental conditions change, not to mention the fact that, more and more, man can change those very environmental conditions himself as well as make physical changes himself by will and not chance. This is an idea more encompassing than the three stages of technological evolution of Radovan Richta. Information is what gives man an edge, and a new kind of edge. It takes little imagination to imagine mankind in the not-too-distant future being in a position such that little save for inner turmoil on a drastic level or a universal shift such as the Big Crunch would threaten his physical existence.

This, then, is how mankind is understood as the end, or final achievement, of evolution: he alone can move the aspect that must change to fit his environment, the aspect of adaptation, outside of himself, through the creation of information. The movement can come to an even more complete fulfillment once it becomes possible for individual members of the human species to fully remove themselves from a biological medium. When this is achieved, along with the developments that it entails, man will no longer have to be concerned with the vast majority of concerns which drove his biological evolution. This will allow for a new type of man to emerge: homo sapiens sapiens will become homo excelsior, man in complete control of his form and his future. He will be truly only limited by his imagination, which needs have no restraints.

Humanity and Willed Evolution

New methods of expression and dissemination of information that have emerged in the recent times that allow his nature as a force of improving the efficiency of his own development to more easily seen. Since intellection and expression of abstract ideas developed in man, this potential existed. From the articulations of philosophers and other thinkers in earlier periods of civilization, e.g. in Ancient Greece, it appears that, in many cases, humanity viewed the universe as something eternal. As soon as the idea of an origin arises, however, the idea that all things might have had a beginning, the idea becomes captivating. It is almost as if man had to develop the idea, and it took on its own internal force.

Once this notion is achieved, man will consider the idea of a variability of form following this origin, if only as a possibility. The idea of things having a beginning and of the state and quality of things being able to change moved from a mere logical possibility, however, to an assumed entity with its own power. Somehow, man arrived at the idea of a progression, that things can be changed for the better in some way. And just as statistical and biological forces cause changes over time in living systems that are beneficial with respect to preservation in some way, so can this happen for man in society, and even personally, in each man’s thoughts. And since man has a will toward his own survival, he can consider altering his own development. This may occur unconsciously, socially, or otherwise at first. But once 2 other factors are present, development can greatly increase:

1. man must decide that he can and will take his own development in hand,


2. man changes the material that development works on from a biological to an informational form.

One of the primary virtues of informational versus biological evolution is that selection in the former can occur independent of individuals. Death has a different meaning in such a context. During the current age, the rate of development in terms of information is much greater than that caused by biological change, mutations, etc. This causes a shift in the understanding of what forms and wholes are, what individuals are, what survival is, etc. Books, articles, networks, even conversations and the ideas that they foster become the new creatures that interact, and survive and reproduce or are destroyed. This change can occur much more readily than its biological analog, and, more importantly, it can be willed. Thus man forms a new medium for his development. It may even be possible for him to transfer all of his content, or at least all of the intellectual content he wishes to preserve, into a non-biological form. He may be able to achieve a manifestation completely based in information and not matter. In this way he could alter his form in any way he wished with extreme rapidity.

The Spatiality of Ideas

It is very interesting to consider the quality of volume that ideas often seem to possess in one’s mind. While this may sound like a rather inchoate notion, I do mean something very definite by it, but I do not yet have a word to express it precisely. I shall give an example instead:
Consider a university or any intellectual institution. When I consider such a thing, it seems to occupy a particular region of some sort of space in my mind. The region corresponds, in some way, to the physical locality of the institution, i.e the region the idea comprises relates to the region representing my self within the totality of my mindspace as my physical person relates to the physical location of the institution. But when I consider more deeply the institution qua intellectual, what was once a volume seems to resolve into points, viz. people and even ideas, the latter melting into some sort of locationless nothing, and the former shrinking to moving points. This seems to be a very subconscious notion, but it is one that I think is socially common. It would be worth considering how our minds connect physical locality with ideas, and more generally how intellectual structure has a spatial element. Perhaps it is merely a matter of association through habit that I connect certain physical locations with certain ideas, but it nevertheless occurs, and has an impact on my method of thought.

If this connection is something more than rote association, however, it would create a whole new level to the database that I have been considering, if the latter is to accurately represent ideas. (I will explain this database in a future post.) There may even be multiple levels to this sort of space, for I do not think that every single one of my ideas have a real or imagined physical correspondent. And even for those that did have a direct physical correspondence, it would not be the sort of correspondent that one could, to speak analogously, point out on a map. It is almost as if this notion requires some sort of fuzziness, i.e. an indistinct notion of location when considering certain ideas.

Perhaps then the database might include a display format to overlap the sense of physicality or even of volume (however distinct either of these may be), of certain ideas, while for others there would be no physical correspondent. Although I do think in the latter case that there is still a spatial sense (since I hold that all ideas, and ideal relations, have a spatial nature) and perhaps this could be meaningfully incorporated with the physical spatiality of ideas possessing such a quality. For simplicity’s sake, this could be displayed as the center of the map being the ideas that do have a physical correspondent, and the edges of it being the ideas which only have and ideal spatiality not a physical one.

In this context it is useful to consider one of Kant's claims within the section on space in the Transcendental Aesthetic of the Critique of Pure Reason (A19 ff). He says that by means of an "outer sense" we present objects as outside of us, within space. The soul’s inner state, however, is intuited through a distinct form: time. But he then claims that space cannot be intuited as something within us.

This seems wholly inconsistent with my experience. My own thoughts are often highly spatial. In fact, my ideas seem to have a spatiality that is either nearly identical to that of physical objects, or one based on it, in a sort of non-temporal extension. It seems odd and almost naïve for Kant to claim that a person is at a single location in space. With respect to certain things this seems true, but that sense should be expanded to include the space of the mind.

The mind as place is an idea which becomes of primary importance when considering how best to represent information, and how human intelligence can be represented.

First contact

This blog is an experiment. I have never kept a blog before, but I think it could be useful for me, as well as interesting. The main reason I am creating it is to document various thoughts that I have had, and new ones which I and others come up with, concerning certain topics, mostly centered on technology and the future of mankind. These will vary widely, as my current interests tend to do. Some of the content may come from various news outlets like Slashdot, as well as others, and some will be my own.

The goal: Become aware of, or create, new and amazing things, record and describe them in a useful and transparent way, ponder their novel and most potentially useful aspects, and record these thoughts for future consideration.