The Tennessee Valley Authority is the nation's largest public power provider serving approximately 9 million consumers in seven southeastern states. The organization also happens to be a big supporter of open-source projects, including Hadoop, a tool designed for deep analysis and transformation of very large data sets.
Earlier this year, the Tennessee Valley Authority (TVA) announced that it open sourced its data system used to collect data from smart grid devices called Phasor measurement units (PMUs). The data collection system is known in the industry as a Super Phasor Data Concentrator (SuperPDC), which can be used to determine the health of a power grid.
The open-source version of the SuperPDC is now called the "OpenPDC." I spoke to both Ritchie Carroll (RC), the project's creator, and Josh Patterson (JP), the person responsible for introducing Hadoop to the project, to discuss what the OpenPDC is and why TVA turned to Hadoop in building the system.
What sort of data volumes are you working with?
RC: Currently there is around 20 TB of archived data, we expect this to grow quickly as a result of the SmartGrid stimulus funding which includes the addition of 850 phasor measurement devices. This may well grow the archive to half a Petabyte within the next few years.
How is this data currently captured and managed? Is any data discarded?
JP: Data is collected directly from field devices at 30 times per second. This data is then time-aligned and processed in real-time--all data gets captured into a binary data file as time-series data for mass processing by Hadoop.
RC: No data is currently discarded, if we get to the point of needing to discard data because of cost--this will be a decision based on weighed importance of collected data. It is likely the data around major events will never be deleted because it will always be valuable for future student researchers. There is also value in being able to go back in time and look for newly discovered event signatures to see how long they might have been occurring.
... Read moreMicrosoft released on Thursday a new position paper, "Privacy in the Cloud Computing Era: A Microsoft Perspective," that includes information about the remote storage and processing of personal information.
Privacy and security concerns continue to be a primary argument that cloud naysayers use against storing data and applications on the Internet. Big IT vendors and service providers like Microsoft and Hewlett-Packard will sooner or later be forced to take the cloud seriously or risk missing out on the whole next wave of IT consumption. And their large enterprise customers will expect them to offer cloud services with the appropriate levels of privacy and security measures in line with their business needs.
The interesting thing about this paper is that Microsoft takes surprisingly minimal responsibility for the data it will manage:
... Read more
(Credit:
IBM)
IBM on Wednesday announced a program designed to help educators and students pursue cloud-computing initiatives and better take advantage of collaboration technology in their studies.
The IBM Cloud Academy, announced at the Educause annual conference, includes a global roster of educational institutions as initial participants. Educause is a nonprofit association whose mission is to advance higher education by promoting the intelligent use of information technology.
IBM will provide the cloud-based infrastructure for the program, with some basic collaboration tools available at the outset. IBM's LotusLive service provides the basis for the new offering. Participants will immediately be able to do some very basic tactical functions on the new system:
- Create working groups on areas of interest to the education industry
- "Jam" on new innovations for clouds in education-related areas with IBM developers
- Work jointly on technical projects across institutions
- Share research findings and exchange new research ideas
Shared research across universities and other higher-learning institutions remains a vital part of technological innovation, but many programs don't have formal tool sets in place. Cloud services are a logical place to run these types of programs, especially as international groups need immediate access to data from their partners.
... Read more
Amazon Associates is the partner program the company uses as part of its affiliate advertising programs, allowing customers to make money advertising Amazon products.
Associates can now simply click a link in the toolbar to send a link (replete with sales-y text) to Twitter as part of their shopping and selling experience. Amazon gets a sale, Twitter gets traffic, and the associate gets revenue share. What could possibly go wrong?
Linking to Amazon or other online retailers is obviously nothing new, though Amazon has been particularly successful in using its link networks for both sales and to garner higher Google rankings for organic advertising.
This new program does introduce an issue related to link fraud, where spammers and scammers leverage URL-shortening services for spam links. Currently there is no way to verify that the link you click actually goes to Amazon. It's a bit surprising that it decided to use an URL-shortener that it doesn't own, though I suppose the network effect of the URLs helps perpetuate the life of the links.
There is also a risk of nondisclosure wherein in Twitter users attempt to push products that offer some kind of gain to them that they don't clearly state to you. While I understand the argument for disclosure on blogs and media in general, Twitter remains a playground for people to post whatever they want. I highly doubt all the celebrities with accounts would bother wasting their precious time if they weren't posting for their own gain.
Interestingly, there is no mention of whether Twitter is an Amazon Associate, suggesting that Twitter won't see any of the revenue share. I'd like to think that they cut a deal that gives them a piece of the pie, but to date we haven't seen Twitter monetize itself too effectively.
Twitter is quickly becoming the flash news vehicle for everything from news alerts to product placement. And based on a very quick review of my Bitly account, Twitter users just love to click on links. But, I still have to wonder if Twitter will ever get beyond its current role as a marketing tool?
As much as Twitter is a powerful communication and social application, it's a relatively simple Web app. As part of a new contest sponsored by Engine Yard, Ruby on Rails developers are going to turn Twitter into their own application server.
The contest asks developers to program the "Worst App Server Technology Ever" (Waste) using Twitter as the message bus. While much of the contest is being done tongue-in-cheek, it's actually an interesting use case to see if a service like Twitter can take the place of a more traditional message bus like IBM MQ series or AMQP (Advanced Message Queuing Protocol).
Contest participants register up to five Twitter handles and code the function that each would perform in a program. When the contest challenge is issued on November 12, participants will have to use at least 10 of the pre-designated Twitter handles (other than their own) as endpoints to perform functions on data sets located at unique URLs. All messages will work through a series of automated public Twitter replies.
This is somewhere between an application server, a social game, the "telephone game" and service-oriented architecture (SOA) where Twitter plays the role of the enterprise service bus and the Twitter API is the broker between data sources. SOA relies on services exposing their functionality other applications and services can read to understand how to utilize those services. In this case, Twitter can be used as an application server in the cloud. (Take that buzzword bingo players.)
The funny thing is that as absurd and comical as this sounded when the Engine Yard guys told me about it, I've started to think about this as a way to possibly achieve a real technological breakthrough. And while I don't think that Twitter will be the "cloud bus," I do think that there is a lot to be learned from applying this type of constraint to a data flow process.
Engine Yard VP of marketing Michael Mullany told me that the contest shows how developers can leverage a relatively straightforward platform in innovative ways. But it's also another example of an interesting marketing effort to use Twitter as the vehicle for one's own benefit. Also, in true open source fashion, developers wind up building new applications based on code written by their peers.
Let's hope Twitter can handle the attention and developers are not greeted by the ever-lurking fail whale. You can check out the contest and learn more details at Engineyard.com
I attended the Virtual Goods Summit on Friday and walked away struggling to figure what topics might be interesting to write about. My net takeaway is that not much has changed in the year that I've been writing about social gaming and virtual goods, with the exception of two facts:
1. Virtual good providers are being lauded as the next big thing to replace advertising
2. There's something weird going on with the ads and offers that have taken over the more traditional banner advertising role
There is no question that virtual goods have become an integral part of social network revenue streams. And the mainstream media has finally started to catch on.
But, I didn't realize the oddities of the way users are being monetized until I attended the event and saw the heavy emphasis not just on monetizing users but on doing so in a way that was transparent and non-intrusive. Theoretically, it's a good idea, but in practice, many of the "offer" providers are purposely or inadvertently running Ponzi schemes.
TechCrunch's Michael Arrington arrived at my second point above and took the theory much further with data that shows many social gaming offers and advertising practices amount to little more than a complicated scam that gets people in the door for free only to take advantage of their lack of understanding of what they've technically agreed to in the various offers.
In short, these games try to get people to pay cash for in game currency so they can level up faster and have a better overall experience. Which is fine. But for users who won't pay cash, a wide variety of "offers" are available where they can get in-game currency in exchange for lead gen-type offers. Most of these offers are bad for consumers because it confusingly gets them to pay far more for in-game currency than if they just paid cash (there are notable exceptions, but the scammy stuff tends to crowd out the legitimate offers). And it's also bad for legitimate advertisers.... Read more
Gamers are not just making purchases to enhance their gaming experience but also selling virtual assets to other players, according to new research from video game market research firm VGMarket.
Sales of virtual goods are expected to reach $1 billion this year and already generate near $4 billion annually in China. But there are some challenges, primarily the fact that once you convert your real money to virtual cash you can't readily get the dough back out.
The research revealed that in-game currency is the most frequently sold digital good from player to player and that two out of three sellers sold in-game currency in the last 12 months, earning a median of $22. PlaySpan, a provider of monetization and payment solutions for games and virtual worlds and sponsor of the research, considers that to be good news as its platform enables game developers to provide player to player marketplaces for their players. In addition, the PlaySpan Marketplace currently provides a secondary market for IMVU players to buy and sell goods as well.
One out of two sellers made a sale in a social network game over the last 12 months and earned a median of $50, while one out of four sellers made a sale in a free-to-play game over the last 12 months, with their median earning being $98, or nearly double that on social networks.
Eric Hartness, chief marketing officer at PlaySpan, told me that the secondary market is a boon for games, adding value, real and perceived, to all players by associating a real world dollar value on their playing time, game accounts, and digital items.
... Read moreInfluence in open-source development communities is earned through years of writing and sharing great code. Perhaps not surprisingly, then, influence in the business side of open source is also gained through sharing expertise, and not necessarily from making mountains of cash.
At least, that's the lesson I take away from MindTouch's inaugural survey of 50 open-source business executives. MindTouch, an open-source collaboration company, has spent the last few months surveying executives within the commercial open-source community, asking them to name the most influential people within the commercial open-source ecosystem.
The result is effectively an all-star list of open-source business executives. The top five are as follows:
- Larry Augustin, CEO, SugarCRM
- Matt Asay, vice president of business development, Alfresco (and fellow CNET blogger)
- Mårten Mickos, entrepreneur-in-residence, Benchmark Capital, and former CEO, MySQL
- Jim Whitehurst, CEO, Red Hat
- Dries Buytaert, co-founder and CTO, Acquia
The full list is available here.
The common theme running through these top-five vote getters is how open they've been with their peers. Larry Augustin sits on several boards of open-source companies, but he also frequently speaks at industry events and has been involved in open source from its inception.
Matt Asay, my friend and fellow CNET blogger, sits on more than 10 open-source advisory boards, chairs the Open Source Business Conference, hosts an informal get-together every year (called Open Source Goat Rodeo--don't ask why), blogs at an unhealthy rate for CNET on open source, and has actively helped a range of aspiring open-source entrepreneurs understand the mechanics of running an open-source business.
Mårten Mickos made the world safe for the $1 billion open-source acquisition, but he has also traveled the globe speaking at open-source events and is very generous with his time, sharing know-how and best practices with other open-source executives.
Jim Whitehurst, breaking the typical Red Hat mold, has been active in industry events, has hosted a range of dinners and other small-scale, intimate events with open-source executives. He is amazingly accessible, given that he has a fast-growing open-source company to run. It's unfortunate that Whitehurst is the only Red Hat executive to make the list; Red Hat should follow his lead and be more permeable to its peers. Its influence would grow accordingly, just as Whitehurst's has.
Finally, there's Dries Buytaert, who blogs frequently on his project, Drupal, but also regularly attends and speaks at industry events. He has also been active behind the scenes, working with other open-source companies to share information on how to optimize community development.
Open-source code becomes valuable when you give it away. The same holds true for open-source business expertise. There are individuals who have made more money than these with open-source software, but in terms of influence, the more you share, the more influential you become.
What do you think? Who else should be on the list? Who influences you?
One of the less appealing aspects of using cloud services is integrating various applications--both those in the cloud and those in your enterprise in an easily manageable way. A practical use case is the ability to use one CRM (customer relationship management) system and a different file storage system, both in the cloud.
So, Friday when I saw that Box.net was directly integrating its cloud-based storage service with Salesforce.com, I saw the confluence of two major trends, cloud storage and integration appear all in one fell swoop.
Salesforce.com has been the leader in cloud services and has consistently offered users ways to integrate other services into their SF.com installation. And Salesforce has also become the prime target for vendors who want to tap into their enormous customer base already using on-demand services.
But, Salesforce is far from infallible with certain aspects, such as storing files somewhat clunky and definitely costly when you start making it your system of record.
With the new offering, Box.net business users will now be able to add a Box.net app to their Salesforce accounts, allowing them to access their documents, media, and other files from directly within their SF.com instance. And they'll have unlimited storage for their files.
... Read moreWith the launch of Windows 7 many signs point to the fact that Microsoft seems to have gotten its operating system engineering in order. That's obviously good news for its OS business unit and also good news for PC manufacturers and software companies that develop for Windows.
A rising tide of Windows adoption is not a bad thing for the technology industry in economic terms but it doesn't yet do a lot to enhance the way we use computers and applications in our every day lives.
The main problem is that Windows 7 reinforces a desktop centric-paradigm for 93 percent of the market and continues to exert a certain level of misguided design principles in the way the system handles data and file structures. And while it's a giant leap forward in terms of customization, visual effects, and security there is a missed opportunity in the cross-border approach of combining the desktop and cloud services.
Cloud-based applications and storage are still so nascent that Microsoft could jump in and usurp much of the power and market share while shoring up its cloud story for the future. Having the dominant desktop landing pad gives Microsoft a huge advantage over upstarts--even Google and Apple, if it can focus on integrating the services.
Here are a few ways Microsoft could assert its dominant desktop position to compete with Google and Amazon as a cloud player:
... Read more
